Alice C. McHardy Credit: yamel photography

Fun is not commonly associated with the task of benchmarking computational tools, but “I think it was lots of fun, actually,” says Alice McHardy, a bioinformatician at Helmholtz Centre for Infection Research in Braunschweig, Germany. Along with others, including Alexander Sczyrba at Bielefeld University and Thomas Rattei from the University of Vienna, she co-organized the first community-based competition devoted to benchmarking software tools—the Critical Assessment of Metagenome Interpretation (CAMI)—and they now present results from the first CAMI challenge.

I think it's amazing what these tiny creatures are capable of.

The aim of running tools through their paces in CAMI is to help labs assess how well software tools perform taxonomic profiling tasks, such as predicting which taxonomic identities and relative abundances are in a given metagenome sample, and how well they assemble and bin genomes by taxa. Forty teams registered for the competition; 19 stayed the course to work with CAMI data sets. These data simulated a microbial community with a specific number of unique genomes and circular elements, particular genome abundances and sequencing read depths. The data were purposefully dissimilar from publicly available data so the teams couldn't cheat.

Some assemblers clearly outperformed others, and taxonomic profilers performed well down to family rank. The tools had some difficulties with detecting circular elements, which might be because the tools are not set up to assess this aspect. “They should, as not only plasmids or viruses can be circular, but also bacterial and archaeal genomes,” says McHardy. She was surprised how well genome binners reconstructed genomes for which no closely related genome data were available. “I am a big fan of these methods now,” she says.

McHardy hopes CAMI continues to grow. This community effort is about helping tool developers and the microbiome community more broadly handle some wild, woolly situations. It's tough to compare results from different papers without an assessment of software performance, she says. Published metagenome analyses often lack well-documented software parameter settings or information about the reference databases used to generate results. Before their experiments can start, labs must sink time into benchmarking tools to select the ones they need for their data.

The idea for CAMI was sparked in 2012 and it was launched in 2014. Preparation for the second challenge is now in the works, as is an automated benchmarking system, says McHardy.

It's the enthusiasm of the community for CAMI that has kept the CAMI team going, she says.

And it takes patience, a skill McHardy has acquired after switching gears into the role of advising others in their research. “It takes much longer than one might expect sometimes initially to get from an idea to the first interesting results,” she says.

When not in the lab, McHardy and her partner tend to do what two-year-olds enjoy. They take their young son swimming and they all like to travel together. One destination this year was the coast of northern Denmark with its grass-covered dunes by the sea. These days she has little time for another creative outlet—her love of dance, including modern and jazz dance.

McHardy completed her PhD in bioinformatics and a postdoctoral fellowship at Bielefeld University. A second postdoc at the IBM T.J. Watson Research Center led to a staff position there. She returned to Germany to direct a group at the Max Planck Institute for Computer Science in Saarbrücken and then chaired algorithmic bioinformatics at Heinrich Heine University. In 2014, she joined Helmholtz Centre for Infection Research, where she leads a computational biology lab.

“During her stay, Alice impressed us frequently with her mastery of non-overlapping fields of science: microbiology, biochemistry, computer science,” says Isidore Rigoutsos, McHardy's postdoctoral advisor at the IBM research center who now directs the Computational Medicine Center at Thomas Jefferson University. McHardy did not like partial answers. “She did not rest until she got to the bottom of things,” he says. “And, time and again she got there, calmly, confidently, always with a smile.”

Although McHardy works on viral pathogens and some human single-cell sequencing data, she says “microbes are currently still my favorite.” She has long worked on microbial genome analysis methods and says, “I think it's amazing what these tiny creatures are capable of.”