There are now many assays to test for the SARS-CoV-2 virus that causes COVID-19. PCR-based assays have turned out to be crucial with SARS-CoV-2. In the United States, for example, emergency use authorizations have been issued by the Food and Drug Administration for different types of tests. Different tests are used in different countries. And sometimes research-only tests are being used.
“If properly validated in a lab, there is no problem whatsoever to use a research-use-only (RUO) test,” says Ghent University researcher Jo Vandesompele. Regulatory authorities indeed sometimes require the use of a regulated test if one is available, “but desperate times require the relaxation of that requirement in my opinion.”
As the world enters the fourth month since the start of the COVID-19 pandemic, “I’m deeply concerned about the rapid escalation and global spread of infection,“ said the director general of the World Health Organization Tedros Adhanom Ghebreyesus at a press briefing in early April. Then came a near-exponential growth of new cases.
Some individuals have “not yet” developed symptoms, and it matters to capture data about these individuals, too, said Maria Van Kerkhove, who is the technical lead of the WHO’s COVID-19 response and who directs emerging diseases and zoonosis at WHO.
Data from China reveals that around 75% individuals categorized as asymptomatic go on to develop symptoms, she said. And one needs to capture data about people who are “PCR-positive” and who do not go on to develop symptoms, she says. “Transmission of this virus is being driven by people who are asymptomatic,” she says. “It’s important to detect the full spectrum of illness and detection through surveillance.”
Developing, validating, troubleshooting assays on a large-scale during a pandemic leads to challenges large and small. At the core of many issues: methods.
The ARTIC network, which stands for Advancing Real-Time Infection Control, a collaboration of several universities in the UK, the US and Belgium devoted to the molecular epidemiology for outbreak response, developed wet-lab and software protocols for characterizing SARS-CoV-2 on nanopore sequencers in eight hours.
The University of Birmingham’s Nicholas Loman is a principal investigator of the ARTIC network. His collaborator at the university, Josh Quick, a UKRI Future Leaders Fellow in the Institute of Microbiology and Infection, designed a tiling scheme to amplify the entire viral genome. But Kentaro Itokawa and colleagues at the National Institute of Infectious Diseases in Tokyo, Japan noticed some issues: it looked like dimers formed between two particular primers, which made them anneal to one another and become less efficient at amplification. The team in Japan developed replacement primers, which the scientists are grateful for and which the ARTIC network quickly adopted, says Loman.
When using nanopore sequencers, says Loman, some labs have begun using the cheaper Flongle flowcell, which cuts down costs. But there is an issue to heed. “If you don't barcode then you don't have the opportunity to run a negative control which can be important, so that's something to be aware of,” he says. That will likely not be a problem for the first sample handled, but “after that you need to be sure you don't have PCR amplicons carrying over.”
World Health Organization
On January 30, 2020, the World Health Organization declared the outbreak of 2019-nCov a Public Health Emergency of International Concern.
In the course of January 2020, the WHO began making molecular assay protocols available from labs willing to share them.
There is a downloadable document here, currently 80 pages long, with protocols, primer and probe sequences from, for example, the China CDC; Institut Pasteur; the US CDC; National Institute of Infectious Diseases, Japan; Charité, Germany; University of Hong Kong; National Institute of Health, Thailand. And there is technical guidance here.
US Centers for Disease Control and prevention
The US Centers for Disease Control and Prevention (CDC) diagnostic kit has faced some reagent issues.
January 31, 2020: COVID-19 is declared a public health emergency in the US. Up to that point, CDC was the only lab permitted to use the CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time Reverse Transcriptase (RT)-PCR Diagnostic Panel. That approach did not scale.
Feb 4, 2020: The US Secretary of Health and Human Services declares a public health emergency. FDA grants emergency use authorization to the CDC for CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time Reverse Transcriptase (RT)-PCR Diagnostic Panel. Now CDC can send it to qualified labs, those in line with Clinical Laboratory Improvement Amendments of 1988 (CLIA) and permitted to perform “high complexity tests.”
February, 2020: A number of public health labs across the country have a variety of issues with the primer/probe sets and inform the CDC. A spokesperson for Integrated DNA Technologies (IDT) tells me that “IDT did not manufacture any component of the lot that CDC Emergency Use Authorization (EUA) testing protocol reported as not working in February.“
Feb 12: Nancy Messonier, who directs the CDC’s National Center for Immunization and Respiratory Diseases says in a press briefing: “...we are re-manufacturing that reagent…”
In this briefing, Messonier stated in response to a question by Reuters reporter Julie Steenhuysen that the test:
…“wasn’t working as expected, specifically some public health labs at states were getting inconclusive results and what that means is that test results were not coming back as false positive or false negatives, but they were being read as inconclusive. “…
…...”In terms of the test problems, it gets a little weedy, but I can give you a little more detail. When a state gets the test kits, they have to verify that it works the same in their lab that it worked at CDC. And when some states were doing this, we received feedback that they weren’t — that it wasn’t working as expected, specifically some public health labs at states were getting inconclusive results and what that means is that test results were not coming back as false positive or false negative, but they were being read as inconclusive.
Now, these were not tests being run on actual clinical specimens from potential patients. These were part of the verification process, and because of that, we are — when we evaluated what the issue is, we think that there might be an issue with one of the three assays and we think that may be one of the reagents wasn’t performing consistently, so it’s a long story to say that we think that the issue at the states can be explained by one reagent that isn’t performing as it should consistently and that’s why we are re-manufacturing that reagent, obviously a state wouldn’t want to be doing this test and using it to make clinical decisions if it isn’t working as well, as perfectly at the state as it is at CDC, so this is part of a normal process and procedure and re-doing the manufacturing is the next step. “…
In response to a question from Erika Edwards, NBC News:
…”yes, all clinical specimens are still being sent to CDC for validation. I think you would expect nothing less from us as obviously the results of this test are so meaningful, and we’ll continue to provide that backstopping frankly even after states are up and running. I would ask that you — that the right language wouldn’t be a problem with the states. It’s a collective problem, so I don’t want this to be seen as something that the states are doing incorrectly. That is certainly not the situation here. This is really part of a normal process and procedure, and, you know, we have the quality control set up specifically to allow us to identify these kinds of problems.“
February 25: According to a spokesperson for IDT, the CDC and FDA contacted IDT to discuss its ability to support the CDC EUA testing protocol. IDT shipped primer and probe kits to the CDC the same day, for delivery on February 26.
March 2: According to IDT, FDA includes IDT lot#0000500383 as the first primer and probe kits qualified under the CDC’s COVID-19 EUA. Subsequently, the CDC has continued to qualify additional lots.
March 15: Letter from FDA chief scientist Denise Hinton to CDC director Robert Redfield. Excerpts:
“Dear Dr. Redfield:
On February 4, 2020, based on a request by the Centers for Disease Control and Prevention (CDC), the Food and Drug Administration (FDA) issued a letter authorizing emergency use of the CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time Reverse Transcriptase (RT)-PCR Diagnostic Panel …”
“On March 5, 2020, FDA received a request from CDC to amend the Emergency Use Authorization (EUA)…. ...“FDA is reissuing the February 4, 2020, letter in its entirety with the amendments incorporated2 to authorize the emergency use of the CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel.“...
“The amendments to the February 4, 2020 letter include:” … “(3) deletion of N3 vial of primer and probe,
(4) use of an alternative fluorescent hydrolysis probe quencher chemistry (ZEN Double-Quenched Probe technology manufactured by Integrated DNA Technologies),
(5) use of commercial manufactured and distributed lots of (i) primer and probe materials identified as acceptable on the CDC website and
(ii) alternative positive control, RP control positive control and extraction control materials listed as acceptable on the CDC website,
(6) use of a separately packaged standalone positive control, and
(7) use of additional extraction methods, extraction reagents and associated instruments.
The authorized Instructions for Use and Fact Sheets also have been updated to incorporate these amendments, where applicable. “ …
“The testing procedure consists of nucleic acid extraction and purification from the human specimen using authorized extraction methods/instruments followed by real time RT-PCR, where the RNA is reverse transcribed into cDNA and then amplified using the primer sets and detected using specific probes.
The real time reverse transcriptase (RT)-PCR is performed on the FDA cleared Applied Biosystems 7500 Fast Dx Real-Time PCR Instrument with SDS 1.4 software, or other authorized instruments or software
The CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel includes the following materials or other authorized materials:
• 2019-nCoV_N1 and 2019-nCoV_N2 vials containing primers and probes that target the nucleocapsid (N) gene and are designed for specific detection of SARS-CoV-2 - either manufactured and/or distributed by CDC, CDC’s International Reagent Resource (IRR), or by commercial manufacturers of lots of primer and probe materials identified as acceptable on the CDC website
• RP vial containing the internal control primers and probes that target the Human RNase P gene – either manufactured and/or distributed by CDC, CDC’s IRR or by commercial manufacturers of lots of primer and probe materials identified as acceptable on the CDC website
• nCoVPC vial containing the SARS-CoV-2 positive control used in the assay – either included in the CDC and IRR distributed kit, provided as a separately packaged control or an alternative commercially manufactured and distributed positive control material listed as acceptable on the CDC website
The CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel also requires the use of additional authorized materials and authorized ancillary reagents that are not included with the test but are commonly used in clinical laboratories and are described in the authorized CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel Instructions for Use.
The CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel requires the following control materials, or other authorized control materials; all controls listed below must generate expected results in order for a test to be considered valid, as outlined in the CDC 2019-nCoV Real-Time RT-PCR Diagnostic Panel Instructions for Use:
• Human Specimen Control (HSC): A human cell culture preparation, negative human specimen material, or an alternative commercially manufactured and distributed extraction control identified as acceptable on the CDC website, used as an extraction control and positive control for the RNase P primer and probe set that is extracted and tested concurrently with each specimen extraction run.
• Positive Control for 2019-nCoV (nCoVPC): Run with each batch of specimens. Monitors for failures of rRT-PCR reagents and reaction conditions.
• No Template Control (NTC): Nuclease-free water included in each run. Monitors for reagent and system contamination.
• RNase P (RP) control in clinical samples: The RP primer and probe set is included in each run to test for human RNase P, which controls for specimen quality and demonstrates that nucleic acid was generated by the extraction process. “
Here are the original primer/probe sets from a CDC document called “2019-Novel Coronavirus (2019-nCoV) Real-time rRT-PCR Panel Primers and Probes.
These are *not* the updated primers and probes for the CDC diagnostic kit.
The PDF describing the CDC diagnostic panel downloads as a document named:
Avian H7 (Eurasian Lineage) Influenza Real-time-RT-PCR Panel Primer Probes.
But the primer probes do not seem to be for Avian flu PCR-based detection.
Perhaps the document is misnamed.
Or the documents are based on the same template document. An online search for H7 CDC primer and probes also yields this document with the protocol for 2019-Novel Coronavirus (2019-nCoV) Real-time rRT-PCR Panel Primers and Probes in the search results.
When Jennifer Rakeman, who heads the New York City Public Health Laboratory, and her colleagues did their initial verification tests of the panel using the CDC’s primer–probe sets called N1, N2, N3, she says there were some spurious results with one of the primer–probe sets in the negative controls and negative specimen samples.
Hopefully we will learn in time from private labs and state labs whether the CDC diagnostic kit continued to work well, despite the initial ‘reagent’ issue, says Tomer Altman, who consults small to medium-sized biotech companies on such areas as human microbiome, databases, biochemical pathways, and data mining. Out of a mixture of curiosity and wanting to help out, he studied the CDC primers. He published his analysis on his blog, on March 3, here.
And I asked him about a few things. For the past year, he has been helping clients with primer design and thought he might be able to make his primers better by studying the CDC’s. “Instead, I found the CDC primers had technical flaws in them that went against the conventional wisdom of how to design primers,” says Altman.
According to Altman, primer-probe pairs N1 and N3 had reverse primers that had high-temperature hairpin loops, he says.
As he notes, the N1 reverse primer for N1 was rejected by primer design software due to a projected hairpin loop.
The N2 primer-probe set had a poly-X run of five 'C' bases in a row on the probe sequence.
The N3 forward primer also has a predicted hairpin loop.
“The Primer3 program, a bedrock bioinformatic program that has been around for over two decades, flagged them as problems,“ he says.
He cautions that he is not a wet-lab biologist but he says that based on his understanding of molecular biology, hairpin loops can lead to inefficient amplification, which in theory could lead to tests that provide indeterminate results, or, in the worst-case scenario, return no results such as a false negative if the patient is infected.
Low sequence complexity, such as the poly-X run, can lead to off-target binding, which can attenuate the fluorescence signal of the probe, as instead of binding specifically to the amplicon, probe oligos are binding elsewhere.
He also found dimers, but they were so low-temperature that it is unlikely that they would lead to problems in any PCR protocol, he says.
He does not know if these technical flaws underpin the mechanism of the reported problems with the CDC diagnostic kits. “I have heard it said that the problems were with ‘reagents’, but that term could mean anything,” he says.
Given these technical flaws these should never have been put into the primer-probe set sequences for a critical test when lives are on the line, says Altman. It is possible that they tested the sequences in vitro and found that they worked great. But he is concerned about the quality control process. “These flaws are known pitfalls in design, and in-silico testing is faster and cheaper than in-vitro testing, so when there's a race against time to get a critical diagnostic developed, why take chances on primer-probe sets with easily-detected defects?”
Vandesompele has not studied Altman’s analysis in detail but says “hairpins may be problematic, but are not always.” One limitation of the analysis is that it only mentions software predictions, and wet-lab data would be needed.
In wet-lab experiments in Mount Sinai’s lab of Benjamin tenOever, postdoctoral fellow Benjamin Nilsson-Payant found the primers “have a lot of background,” tenOever says.
March 13, 2020: He tweeted this information and the gel. Immediate discussion ensued here.
What matters in clinical testing of a pathogen are the positive and negative predictive value, how sure can one be that a positive or negative test is informative about a person having or not having an illness. Other considerations are cost and turnaround time, says Vandosompele.
As Nicholas Loman from the University of Birmingham says, “It was disappointing the CDC came out with a pretty poor primer design first off, when there were better ones available from other groups, such as the Hong Kong set,” he says, referring to primers developed at the University of Hong Kong.
“Indeed, the primer sequences are important, but other aspects as well,” says Vandesompele. Those include: sample preparation, such as nucleic acid purification, the reverse transcription, and the PCR reaction conditions themselves such as the concentration of primers, enzyme, buffer, temperature cycling protocols and other factors. The reverse transcription step, which is needed since this is an RNA virus, is “an often overlooked, but critical step” that amongst others determines analytical sensitivity.
RT-PCR tests are suitable for early screening and sequencing-based kits are used for concurrent detection of infection or to confirm results in complicated infections, says Weijun Chen, chief scientist of infectious disease at BGI Research whose comments were relayed through a spokesperson. “The same types of kits have different potency,” says Chen. More sensitive kits should be selected first, but the cost may be higher. In late January, BGI received emergency approval for its RT-PCR kit from China’s National Medical Products Administration, and in late March it received an EUA from the FDA.
Tests need to have been tested in the “real world,” they have to generate results quickly and be scalable also for population-wide screening, says Chen. And “you want to use what you already have in the lab to start the test right away, not to wait to buy a new machine,” he says.
Chinese authorities have deployed several kinds of RT-PCR-based tests, including BGI’s kit, which is one of the earliest tests developed and was also used clinically in China, says Chen. BGI was also one of the first third-party labs to detect the novel coronavirus. As of early April, the company had manufactured and distributed 2.1 million tests to over 70 countries and had completed 520,000 tests in their labs. The company can manufacture 2 million tests a day, says Chen.
As Chen explains, the BGI RT-PCR test includes a control and it was validated in terms of specificity by cross-testing against, for example, common respiratory viruses, bacterial cultures, clinical samples. Sensitivity was tested with samples from patients with confirmed COVID-19 disease and to find the minimum detection threshold the team used serial dilutions of clinically positive samples with known concentrations, and repeated experiments at least 20 times. They also evaluated the shelf life of the test reagents and overall test kit stability.
BGI designed a lab in Wuhan, called Huo-Yan or ‘Fire Eye' Laboratory that was built, he says, in five days and it is capable of performing 10,000 tests a day. It includes instruments from a number of vendors, such as MGI’s automated sample prep system for RNA extraction and reagent mixing and a separate MGI platform for metagenomic analysis to analyze secondary infection in severely ill patients.
This lab in Wuhan was built in five days, according to Weijun Chen, chief scientist of infectious disease at BGI Research whose comments were relayed through a spokesperson. 10,000 dests a day can be performed in this facility. (Photo: BGI)
Beyond this lab, BGI set up 12 other labs across China for coronavirus detection. The company is also collaborating with labs in several countries to set up labs there, he says. The company’s solution is standardized and scalable and the teams have experience with outbreaks such as SARS, avian flu, and Ebola.
The company is in touch with WHO about its PCR-based test, is conducting test comparisons and giving recommendations, says Chen. His company is in touch with many labs in Europe, the US and Canada. There is no standard World Health Organization test but WHO mentions some recommendations. As of early April, the company’s daily manufacturing capacity is two million tests per day, Chen says.
For primer design, the company scientists analyzed the viral genome, selected specific genomic regions for in silico primer probe design and checked them for hairpin or dimer structures. They then adjusted the probes to avoid such issues, says Chen. The company has built a metagenomics approach for discovery of unknown pathogens in complex clinical samples. Among the new viruses they have detected is a new bunyavirus. For this research, the company’s uses its high-throughput sequencer and viral genome assembly methods.
The WHO is evaluating and comparing the performance of molecular tests and imunoassays in collaboration with institutes such as Hôpitaux Universitaires Genève and others.
The WHO had invited submissions and received 220 of them from developers of nucleic acid assays, and in a separate call related to immunoassays, received nearly 100 of those. As soon as the evaluation is completed, the WHO says it will make the results publicly available.