How to share data with a statistician

How to share data with a statistician This is a guide for anyone who needs to share data with a statistician. The target audiences I have in mind are: Scientific collaborators who need statisticians to analyze data for them Students or postdocs in scientific disciplines looking for consulting advice Junior statistics students whose job it is to collate/clean data sets The goals of this guide are to provide some instruction on the best way to share data to avoid the most common pitfalls and sources of delay in the transition from data collection to data analysis. The Leek group works with a large number of collaborators and the number one source of variation in the speed to results is the status of the data when they arrive at the Leek group. Based on my conversations with other statisticians this is true nearly universally. My strong feeling is that statisticians should be able to handle the data in whatever state they arrive. It is important to see the raw data, understand the steps in the processing pipeline, and be able to incorporate hidden sources of variability in one's data analysis. On the other hand, for many data types, the processing steps are well documented and standardized. So the work of converting the data from raw form to directly analyzable form can be performed before calling on a statistician. This can dramatically speed the turnaround time, since the statistician doesn't have to work through all the pre-processing steps first. What you should deliver to the statistician For maximum speed in the analysis this is the information you should pass to a statistician: The raw data. A tidy data set A code book describing each variable and its values in the tidy data set. An explicit and exact recipe you used to go from 1 -> 2,3 Let's look at each part of the data package you will transfer. The raw data It is critical that you include the rawest form of the data that you have access to. Here are some examples of the raw form of data: The strange binary file your measurement machine spits out The unformatted Excel file with 10 worksheets the company you contracted with sent you The complicated JSON data you got from scraping the Twitter API The hand-entered numbers you collected looking through a microscope You know the raw data is in the right format if you: Ran no software on the data Did not manipulate any of the numbers in the data You did not remove any data from the data set You did not summarize the data in any way If you did any manipulation of the data at all it is not the raw form of the data. Reporting manipulated data as raw data is a very common way to slow down the analysis process, since the analyst will often have to do a forensic study of your data to figure out why the raw data looks weird. The tidy data set The general principles of tidy data are laid out by Hadley Wickham in this paper and this video. The paper and the video are both focused on the R package, which you may or may not know how to use. Regardless the four general principles you should pay attention to are: Each variable you measure should be in one column Each different observation of that variable should be in a different row There should be one table for each "kind" of variable If you have multiple tables, they should include a column in the table that allows them to be linked While these are the hard and fast rules, there are a number of other things that will make your data set much easier to handle. First is to include a row at the top of each data table/spreadsheet that contains full row names. So if you measured age at diagnosis for patients, you would head that column with the name AgeAtDiagnosis instead of something like ADx or another abbreviation that may be hard for another person to understand. Here is an example of how this would work from genomics. Suppose that for 20 people you have collected gene expression measurements with RNA-sequencing. You have also collected demographic and clinical information about the patients including their age, treatment, and diagnosis. You would have one table/spreadsheet that contains the clinical/demographic information. It would have four columns (patient id, age, treatment, diagnosis) and 21 rows (a row with variable names, then one row for every patient). You would also have one spreadsheet for the summarized genomic data. Usually this type of data is summarized at the level of the number of counts per exon. Suppose you have 100,000 exons, then you would have a table/spreadsheet that had 21 rows (a row for gene names, and one row for each patient) and 100,001 columns (one row for patient ids and one row for each data type). If you are sharing your data with the collaborator in Excel, the tidy data should be in one Excel file per table. They should not have multiple worksheets, no macros should be applied to the data, and no columns/cells should be highlighted. Alternatively share the data in a CSV or TAB-delimited text file. The code book For almost any data set, the measurements you calculate will need to be described in more detail than you will sneak into the spreadsheet. The code book contains this information. At minimum it should contain: Information about the variables (including units!) in the data set not contained in the tidy data Information about the summary choices you made Information about the experimental study design you used In our genomics example, the analyst would want to know what the unit of measurement for each clinical/demographic variable is (age in years, treatment by name/dose, level of diagnosis and how heterogeneous). They would also want to know how you picked the exons you used for summarizing the genomic data (UCSC/Ensembl, etc.). They would also want to know any other information about how you did the data collection/study design. For example, are these the first 20 patients that walked into the clinic? Are they 20 highly selected patients by some characteristic like age? Are they randomized to treatments? A common format for this document is a Word file. There should be a section called "Study design" that has a thorough description of how you collected the data. There is a section called "Code book" that describes each variable and its units. How to code variables When you put variables into a spreadsheet there are several main categories you will run into depending on their data type: Continuous Ordinal Categorical Missing Censored Continuous variables are anything measured on a quantitative scale that could be any fractional number. An example would be something like weight measured in kg. Ordinal data are data that have a fixed, small (< 100) number of levels but are ordered. This could be for example survey responses where the choices are: poor, fair, good. Categorical data are data where there are multiple categories, but they aren't ordered. One example would be sex: male or female. Missing data are data that are missing and you don't know the mechanism. You should code missing values as NA. Censored data are data where you know the missingness mechanism on some level. Common examples are a measurement being below a detection limit or a patient being lost to follow-up. They should also be coded as NA when you don't have the data. But you should also add a new column to your tidy data called, "VariableNameCensored" which should have values of TRUE if censored and FALSE if not. In the code book you should explain why those values are missing. It is absolutely critical to report to the analyst if there is a reason you know about that some of the data are missing. You should also not impute/make up/ throw away missing observations. In general, try to avoid coding categorical or ordinal variables as numbers. When you enter the value for sex in the tidy data, it should be "male" or "female". The ordinal values in the data set should be "poor", "fair", and "good" not 1, 2 ,3. This will avoid potential mixups about which direction effects go and will help identify coding errors. Always encode every piece of information about your observations using text. For example, if you are storing data in Excel and use a form of colored text or cell background formatting to indicate information about an observation ("red variable entries were observed in experiment 1.") then this information will not be exported (and will be lost!) when the data is exported as raw text. Every piece of data should be encoded as actual text that can be exported. The instruction list/script You may have heard this before, but reproducibility is kind of a big deal in computational science. That means, when you submit your paper, the reviewers and the rest of the world should be able to exactly replicate the analyses from raw data all the way to final results. If you are trying to be efficient, you will likely perform some summarization/data analysis steps before the data can be considered tidy. The ideal thing for you to do when performing summarization is to create a computer script (in R, Python, or something else) that takes the raw data as input and produces the tidy data you are sharing as output. You can try running your script a couple of times and see if the code produces the same output. In many cases, the person who collected the data has incentive to make it tidy for a statistician to speed the process of collaboration. They may not know how to code in a scripting language. In that case, what you should provide the statistician is something called pseudocode. It should look something like: Step 1 - take the raw file, run version 3.1.2 of summarize software with parameters a=1, b=2, c=3 Step 2 - run the software separately for each sample Step 3 - take column three of outputfile.out for each sample and that is the corresponding row in the output data set You should also include information about which system (Mac/Windows/Linux) you used the software on and whether you tried it more than once to confirm it gave the same results. Ideally, you will run this by a fellow student/labmate to confirm that they can obtain the same output file you did. What you should expect from the analyst When you turn over a properly tidied data set it dramatically decreases the workload on the statistician. So hopefully they will get back to you much sooner. But most careful statisticians will check your recipe, ask questions about steps you performed, and try to confirm that they can obtain the same tidy data that you did with, at minimum, spot checks. You should then expect from the statistician: An analysis script that performs each of the analyses (not just instructions) The exact computer code they used to run the analysis All output files/figures they generated. This is the information you will use in the supplement to establish reproducibility and precision of your results. Each of the steps in the analysis should be clearly explained and you should ask questions when you don't understand what the analyst did. It is the responsibility of both the statistician and the scientist to understand the statistical analysis. You may not be able to perform the exact analyses without the statistician's code, but you should be able to explain why the statistician performed each step to a labmate/your principal investigator. Contributors Jeff Leek - Wrote the initial version. L. Collado-Torres - Fixed typos, added links. Nick Reich - Added tips on storing data as text.

A forever friend

"A friend walk in when the rest of the world walks out." Sometimes in life, You find a special friend; Someone who changes your life just by being part of it. Someone who makes you laugh until you can't stop; Someone who makes you believe that there really is good in the world. Someone who convinces you that there really is an unlocked door just waiting for you to open it. This is Forever Friendship. when you're down, and the world seems dark and empty, Your forever friend lifts you up in spirits and makes that dark and empty world suddenly seem bright and full. Your forever friend gets you through the hard times, the sad times, and the confused times. If you turn and walk away, Your forever friend follows, If you lose you way, Your forever friend guides you and cheers you on. Your forever friend holds your hand and tells you that everything is going to be okay. And if you find such a friend, You feel happy and complete, Because you need not worry, Your have a forever friend for life, And forever has no end.

Promise

In 1989 an 8.2 earthquake almost flattened America, killing over 30,000 people in less than four minutes. In the midst of utter devastation and chaos, a father left his wife safely at home and rushed to the school where his son was supposed to be, only to discover that the building was as flat as a pancake. After the unforgettably initial shock, he remembered the promise he had made to his son: "No matter what, I’ll always be there for you!" And tears began to fill his eyes. As he looked at the pile of ruins that once was the school, it looked hopeless, but he kept remembering his commitment to his son. He began to direct his attention towards where he walked his son to class at school each morning. Remembering his son s classroom would be in the back right corner of the building; he rushed there and started digging through the ruins. As he was digging, other helpless parents arrived, clutching their hearts, saying: "My son!" "My daughter!" Other well meaning parents tried to pull him off what was left of the school, saying: "It s too late! They’re all dead! You can’t help! Go home! Come on, face reality, there s nothing you can do!" To each parent he responded with one line: "Are you going to help me now?" And then he continued to dig for his son, stone by stone. The fire chief showed up and tried to pull him off the school s ruins saying, "Fires are breaking out, explosions are happening everywhere. You’re in danger. We’ll take care of it. Go home." To which this loving, caring American father asked, "Are you going to help me now?" The police came and said, "You’re angry, anxious and it s over. You’re endangering others. Go home. We’ll handle it!" To which he replied, "Are you going to help me now?" No one helped.Courageously he went on alone because he needed to know for himself: "Is my boy alive or is he dead?" He dug for eight hours...12 hours...24 hours...36 hours...then, in the 38th hour, he pulled back a large stone and heard his son s voice. He screamed his son s name, "ARMAND!" He heard back, "Dad!?! It s me, Dad! I told the other kids not to worry. I told them that if you were alive, you d save me and when you saved me, they d be saved. You promised, No matter what happens, I’ll always be there for you! You did it, Dad!" "What s going on in there? How is it?" the father asked. "There are 14 of us left out of 33, Dad. We’re scared, hungry, thirsty and thankful you re here. When the building collapsed, it made a triangle, and it saved us." "Come out, boy!" "No, Dad! Let the other kids out first, cause I know you ll get me! No matter what happens, I know you’ll always be there for me!"

Run through the rain

She had been shopping with her Mom in Wal-Mart. She must have been 6 years old, this beautiful brown haired, freckle-faced image of innocence. It was pouring outside. The kind of rain that gushes over the top of rain gutters, so much in a hurry to hit the Earth, it has no time to flow down the spout. We all stood there under the awning and just inside the door of the Wal-Mart. We all waited, some patiently, others irritated, because nature messed up their hurried day. I am always mesmerized by rainfall. I get lost in the sound and sight of the heavens washing away the dirt and dust of the world. Memories of running, splashing so carefree as a child come pouring in as a welcome reprieve from the worries of my day. Her voice was so sweet as it broke the hypnotic trance we were all caught in, “Mom, let's run through the rain." she said. "What?" Mom asked. "Let's run through the rain!" She repeated. "No, honey. We'll wait until it slows down a bit." Mom replied. This young child waited about another minute and repeated: "Mom, let's run through the rain." "We'll get soaked if we do." Mom said. "No, we won't, Mom. That's not what you said this morning," the young girl said as she tugged at her Mom's arm." "This morning? When did I say we could run through the rain and not get wet?" "Don't you remember? When you were talking to Daddy about his cancer, you said, If God can get us through this, he can get us through anything!" The entire crowd stopped dead silent. I swear you couldn't hear anything but the rain. We all stood silently. No one came or left in the next few minutes. Mom paused and thought for a moment about what she would say. Now some would laugh it off and scold her for being silly. Some might even ignore what was said. But this was a moment of affirmation in a young child's life. Time when innocent trust can be nurtured so that it will bloom into faith. "Honey, you are absolutely right. Let's run through the rain. If get wet, well maybe we just needed washing." Mom said. Then off they ran. We all stood watching, smiling and laughing as they darted past the cars and. They held their shopping bags over their heads just in case. They got soaked. But they were followed by a few who screamed and laughed like children all the way to their cars. And yes, I did. I ran. I got wet. I needed washing.Circumstances or people can take away your material possessions, they can take away your money, and they can take away your health. But no one can ever take away your precious memories. So, don't forget to make time and take the opportunities to make memories every day! To everything there is a season and a time to every purpose under heaven. I hope you still take the time to run through the rain.

A special letter

Dear World: My son starts school today. It's going to be strange and new to him for a while, and I wish you would sort of treat him gently. You see, up to now, he's been king of the roost. He's been boss of the backyard. I have always been around to repair his wounds, and to soothe his feelings. But now--things are going to be different. This morning, he's going to walk down the front steps, wave his hand and start on his great adventure that will probably include wars and tragedy and sorrow. To live his life in the world he has to live in will require faith and love and courage. So, World, I wish you would sort of take him by his young hand and teach him the things he will have to know. Teach him - but gently, if you can. Teach him that for every scoundrel there is a hero; that for every crooked politician there is a dedicated leader; that for every enemy there is a friend. Teach him the wonders of books. Give him quiet time to ponder the eternal mystery of birds in the sky, bees in the sun, and flowers on the green hill. Teach him it is far more honorable to fail than to cheat. Teach him to have faith in his own ideas, even if everyone tells him they are wrong. Teach him to sell his brawn and brains to the highest bidder, but never to put a price on his heart and soul. Teach him to close his ears to a howling mob...and to stand and fight if he thinks he's right. Teach him gently, World, but don't coddle him, because only the test of fire makes fine steel. This is a big order, World, but see what you can do. He's such a nice little fellow.

zabbix解决中文乱码

解决中文乱码问题 1.从windows下控制面板->字体->选择一种中文字库例如“楷体” 2.把它拷贝到zabbix的web端的fonts目录下例如：/var/www/html/zabbix/fonts，并且把TTF后缀改为ttf 3.修改zabbix的web端/include/defines.inc.php 点击(此处)折叠或打开 //define('ZBX_FONT_NAME', 'DejaVuSans'); define('ZBX_FONT_NAME', 'SIMKAI'); //define('ZBX_GRAPH_FONT_NAME', 'DejaVuSans'); // font file name define('ZBX_GRAPH_FONT_NAME', 'SIMKAI'); // font file name 其中SIMKAI为字库名字,不包含ttf后缀

Linux非root用户如何使用80端口启动程序

默认情况下Linux的1024以下端口是只有root用户才有权限占用，我们的tomcat，apache，nginx等等程序如果想要用普通用户来占用80端口的话就会抛出java.net.BindException: Permission denied:80的异常。 解决办法有两种： 1.使用非80端口启动程序，然后再用iptables做一个端口转发。 iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080 用root用户直接去执行就可以了！ 2.假设我们需要启动的程序是nginx，那么这么做也可以达到目的。 一开始我们查看nginx的权限描述： www.2cto.com -rwxr-xr-x 1 nginx dev 2408122 Sep 5 16:01 nginx 这个时候必然是无法正常启动的。 首先修改文件所属用户为root： chown root nginx 然后再加上s权限： chmod u+s nginx 再次查看权限描述的时候： -rwsr-xr-x 1 root root 2408122 Sep 5 16:01 nginx 这样就能启动了。

监控服务器mysql变为只读文件问题解决

监控服务器的mysql不知道什么原因，突然停止，造成文件只读， 解决办法 1.重启看是否可以修复（很多机器可以） 2.使用用fsck – y 来修复文件系统 3.若，在进行修复的时候有的分区会报错，重新启动系统问题依旧 查看下分区结构 [root@localhost mobile]# more /etc/fstab [root@localhost ~]# more /proc/mounts [root@localhost ~]# mount /dev/sda3 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (ro) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 查看ro挂载的分区,如果发现有ro，就重新mount umount /dev/sda1 mount /dev/sda1 /boot 如果发现有提示“device is busy”，找到是什么进程使得他busy fuser -m /boot 将会显示使用这个模块的pid fuser -mk /boot 将会直接kill那个pid 然后重新mount即可。 4.直接remount，命令为 [root@localhost ~]# mount -o rw,remount /boot ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ linux系统重启或无故变为只读造成网站无法正常访问的简单临时的做法： 一 1、mount: 　　用于查看哪个模块输入只读，一般显示为： 　　/dev/hda1 on / type ext3 (rw) 　　none on /proc type proc (rw) 　　usbdevfs on /proc/bus/usb type usbdevfs (rw) 　　none on /dev/pts type devpts (rw,gid=5,mode=620) 　　/dev/hda5 on /home type ext3 (rw) 　　none on /dev/shm type tmpfs (rw) 　　/dev/hda2 on /usr/local type ext3 (rw) 　　/dev/nb1 on /EarthView/RAW type ext3 (ro)（变为只读了） 　　2、如果发现有ro，就重新mount，或者umount以后再remount 　　3、umount /dev/nb1 　　如果发现有提示“device is busy”，找到是什么进程使得他busy 　　fuser -m /mnt/data 将会显示使用这个模块的pid 　　fuser -mk /mnt/data 将会直接kill那个pid 　　然后重新mount即可。 　　4、还有一种方法是直接remount，命令为 　　mount -o rw,remount /mnt/data 二 具体深入的做法，情况不同可以自行选择： 服务器/var/log/messages报错 ： end_request: I/O error, dev sda, sector 122194293 Buffer I/O error on device sda1, logical block 446493 lost page write due to I/O error on sda1 下面是整个处理全过程 [root@php5 ~]# fdisk -lu #第一步 ：找出本地扇片所在的分区。 Disk /dev/sda: 73.4 GB, 73407868928 bytes 255 heads, 63 sectors/track, 8924 cylinders, total 143374744 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/sda1 * 63 4096574 2048256 83 Linux /dev/sda2 4096575 75778604 35841015 83 Linux /dev/sda3 75778605 129034079 26627737+ 83 Linux /dev/sda4 129034080 143364059 7164990 5 Extended /dev/sda5 129034143 139267484 5116671 83 Linux /dev/sda6 139267548 143364059 2048256 82 Linux swap [root@php5 ~]# tune2fs -l /dev/sda3 |grep "Block size" #找到block大小。 Block size: 4096 (122194293-75778605)*512/4096 =528691 利用公式算出逻辑块地址 b = (int)((L-S)*512/B) [root@php5 ~]# debugfs debugfs 1.35 (28-Feb-2004) debugfs: open /deb/sda3 /deb/sda3: No such file or directory while opening filesystem debugfs: open /dev/sda3 debugfs: icheck 582391 Block Inode number 582391 277584 debugfs: ncheck 277584 Inode Pathname 277584 /users/inn.net.cn/data/upload/download/innshow004.rar debugfs: quit [root@php5 ~]#dd if=/dev/zero of=/dev/sda1 bs=4096 count=1 seek=582391 #找到这个快的文件之后，需要做好备份，我们强制把它设置为0字节。 [root@php5 ~]# sync

windows mysql 自动备份的几种方法

基于之前的文章方法，加入批处理命令即可实现自动备份。只是由于批处理命令中对于备份文件的名字按照时间命名比较特别，所以特别整理一文。 1、复制date文件夹备份 ============================ 假想环境： MySQL 安装位置：C:\MySQL 论坛数据库名称为：bbs 数据库备份目的地：C:\db_bak\ ============================ 新建db_bak.bat，写入以下代码 *******************************Code Start***************************** net stop mysql xcopy c:\mysql\data\bbs\*.* c:\db_bak\bbs\%date:~0,10%\ /S /I net start mysql *******************************Code End ***************************** 然后使用Windows的“计划任务”定时执行该批处理脚本即可。（例如：每天凌晨3点执行back_db.bat） 解释：备份和恢复的操作都比较简单，完整性比较高，控制备份周期比较灵活，例如，用%date:~0,10%。此方法适合有独立主机但对mysql没有管理经验的用户。缺点是占用空间比较多，备份期间mysql会短时间断开（例如：针对30M左右的数据库耗时5s左右）,针对%date:~0,10%的用法参考 。 2、mysqldump备份成sql文件 ============== 假想环境： MySQL 安装位置：C:\MySQL 论坛数据库名称为：bbs MySQL root 密码：123456 数据库备份目的地：D:\db_backup\ 脚本： rem *******************************Code Start***************************** @echo off set "Ymd=%date:~,4%%date:~5,2%%date:~8,2%" C:\MySQL\bin\mysqldump --opt -u root --password=123456 bbs > D:\db_backup\bbs_%Ymd%.sql @echo on rem *******************************Code End***************************** 将以上代码保存为backup_db.bat 然后使用Windows的“计划任务”定时执行该脚本即可。（例如：每天凌晨5点执行back_db.bat） 说明：此方法可以不用关闭数据库，并且可以按每一天的时间来名称备份文件。 通过%date:~5,2%来组合得出当前日期，组合的效果为yyyymmdd,date命令得到的日期格式默认为yyyy-mm-dd(如果不是此格式可以通过pause命令来暂停命令行窗口看通过%date:~,20%得到的当前计算机日期格式)，所以通过%date:~5,2%即可得到日期中的第五个字符开始的两个字符，例如今天为2009-02-05,通过%date:~5,2%则可以得到02。（日期的字符串的下标是从0开始的） 3、利用WinRAR对MySQL数据库进行定时备份。 对于MySQL的备份，最好的方法就是直接备份MySQL数据库的Data目录。下面提供了一个利用WinRAR来对Data目录进行定时备份的方法。 首先当然要把WinRAR安装到计算机上。 将下面的命令写入到一个文本文件里 *******************************Code Start***************************** net stop mysql c:\progra~1\winrar\winrar a -ag -k -r -s d:\mysql.rar d:\mysql\data net start mysql *******************************Code End***************************** 保存，然后将文本文件的扩展名修改成CMD。进入控制面版，打开计划任务，双击“添加计划任务”。在计划任务向导中找到刚才的CMD文件，接着为这个任务指定一个运行时间和运行时使用的账号密码就可以了。 这种方法缺点是占用时间比较多，备份期间压缩需要时间，mysql断开比第一种方法更多的时间，但是对于文件命名很好。

如何禁止cron发用户邮件

经常会遇到这样的问题，登陆系统或者敲命令时，系统总会提示： You have new mail in /var/spool/mail/root 有时很烦，具体产生的原因如下： cron中执行的脚本有输出导致的。很多人写脚本都不考虑标准输出和错误输出如何处理，这样一旦有输出，cron就会把输出内容发邮件给当前用户。很多网上说的停止MTA（sendmail or postfix）等是不管用的，删除sendmail命令也不行。 比较普通的处理方法就是在每条cron后面添加如下方法进行屏蔽。 点击(此处)折叠或打开 >/dev/null 2>&1. OR &> /dev/null 如： */2 * * * * /usr/local/sbin/dog_lighttpd.sh >/dev/null 2>&1 但是有时候还不够方便，比如刚接手的一个项目，里面已经写了300多条cron了，即使写脚本添加屏蔽也比较烦。 那么还有更简单的方法： 直接：crontab -e 在第一行添加：MAILTO="" 原来在/etc/crontab中定义了收件人，所以我们把收件人滞空就可以了。 其实最根本的方法还是大家养成脚本规范的好习惯。