Application Note
AI-enabled automated compound screening for toxicity effects using healthy human intestinal organoids
- Reliable and reproducible: Assay-ready 3D duodenum/intestinal organoids offer easy-to-use reproducible models for intestinal toxicity and drug testing
- Consistency and scalability: Automated workflows for culturing intestinal organoids ensure consistency and scalability for compound toxicity testing.
- Enhanced imaging: High-resolution, confocal imaging of organoids captures phenotypic changes in organoids with enhanced speed and clarity.
- Accurate data analysis: Machine learning tools facilitate image analysis enabling robust classification of organoid phenotypes and providing sensitive and accurate quantification of toxicity effects
Krishna Macha, Auguste Kersulyte, Zhisong Tong,
Giusy Tornillo, Oksana Sirenko | Molecular Devices, LLC
Introduction
A major limitation in the clinical use of anti-cancer drugs is their off-target toxicity, particularly to the intestinal epithelium. Damage to intestinal cells is one of the most common and dose-limiting side effects of chemotherapy, often restricting therapeutic efficacy. To better predict such adverse effects, it is essential to develop in vitro models that would accurately reflect the physiological conditions of human tissues and facilitate pre-screening drugs and drug candidates for toxicity. While conventional two-dimensional (2D) cell culture systems are widely used for toxicity testing, they often fail to replicate the complex structure, cellular interactions, and functional responses of native tissues. In contrast, human intestinal organoids offer a three-dimensional (3D) culture model derived from pluripotent or adult stem cells that selforganize into miniaturized structures resembling the native intestine. These organoids recapitulate key features of intestinal architecture and function, providing a more predictive platform for drug screening. Notably, studies have demonstrated a high degree of correlation between patient responses and those of patient-derived organoids, supporting their application in personalized medicine and preclinical toxicity evaluation.1,2
In this study, we established an automated platform to assess compound-induced intestinal toxicity using human intestinal organoids. Assay-ready organoids were cultured in Matrigel® domes using the CellXpress.ai® Automated Cell Culture System, which automated human intestinal organoid culture, compound addition, and staining. For compound testing, organoids were exposed to eight known drugs that cause intestinal toxicity.
Toxicity evaluation was performed using high-content imaging. Image-based high-content screening (HCS) is a potent drug discovery strategy that measures and characterizes phenotypic changes caused by compounds or other perturbances through the quantification of image-based features that describe changes in cells and organoids. For the endpoint imaging assays, we used the ImageXpress® HCS.ai High-Content Screening System, which features advanced confocal optics for high-resolution imaging and artificial intelligence (AI)- enabled complex image analysis. Following treatment, organoids were stained with fluorescent markers for nuclei, mitochondria, viability, and cytoskeletal integrity, and imaged at 10X magnification. Image analysis was performed using IN Carta® Image Analysis Software, which automated segmentation and feature extraction across multiple parameters. Machine learning (ML) algorithms were applied to quantify phenotypic responses. An initial unsupervised clustering defined distinct phenotype groups, followed by supervised classification to categorize organoids as intact or damaged. This approach is wellsuited for automating toxicity assessment workflows, substantially minimizing manual cell handling while improving both productivity and scalability of the assay. The use of AI-driven data analysis streamlines complex analytical processes, ensuring efficient, consistent, and high-throughput compound evaluation.
Methods
Organoid culture, treatment, staining
3D Ready™ human normal duodenal organoids1–2 (Molecular Devices, San Jose, California, OES-DP41N2-CXP1) were cultured in Matrigel domes using IntestiCult™ Organoid Growth medium (Human) (StemCell Technologies, Vancouver, Canada, #06010). During organoid culture, automated media exchanges and monitoring by transmitted light imaging were performed every 48 and 24 hours, respectively. For toxicity evaluation, assay-ready organoids were seeded into 96-well plates (U-bottom Corning), in 50% Matrigel domes, 18 μL per dome (50/μL, 900–1000 per well). Each dome contained approximately 75 fully grown organoids. Organoids were plated using the CellXpress.ai Automated Cell Culture System. Compounds were added to organoids after 48h in culture. Compound treatments were applied using a 7-point dilution range, 5X concentrations, starting from 100 μM concentration, except for staurosporine, which started from 10 μM concentration. All the compounds were from Sigma. Organoids were cultured with compounds for 72 hours. After compound treatments, organoids were stained with Hoechst (5 µg/mL) and MitoTracker orange (500 nM), and Calcein AM (2 µM) followed by imaging on the ImageXpress HCS.ai system for live imaging. All the dyes were from Thermo Scientific.
Imaging organoids
Organoids were imaged using the ImageXpress HCS.ai system with the standard confocal option (60 µm pinhole) in three fluorescent channels: DAPI, FITC, TRITC, and 10X magnification. 2x2 sites per well were taken at 10X magnification to cover organoids in the center of the well. Z-stacks of 16 images were taken at 8μm intervals, covering a z-range of 160 μm. Maximum projection 2D images were used for analysis.
High-content image analysis
Image analysis was completed on 2D maximum projection images using IN Carta software. The first method used the Custom Module Editor (CME) tool in the IN Carta software to create a multi-step protocol that defined organoids as “blobs” using the DAPI channel (nuclear stain). The number of organoids, average organoid area, and average organoid fluorescent intensities for Hoechst, Calcein AM and Mitotracker were measured using DAPI, FITC and TRITC filter sets respectively. First, a Gaussian filter was used to blur the Hoechst signal to facilitate the segmentation of organoids. Next, nuclei were segmented and used to define cells. Cells were scored as positive or negative depending on the signal intensities for Calcein AM or MitoTracker. Thresholds for positive and negative cell scoring were set empirically using control (untreated) samples and samples treated with a toxic compound. Cells that were scored positive for Calcein AM or Mitotracker were defined as intact or live. Positive and negative cells per organoid were counted, and the average area and average intensity of positive cells were measured. After analysis, concentration dependencies for different readouts were plotted as a 4-parametric curve fit (for 0.06–100 µM concentration range) to calculate IC50s for compound toxicity effects. SoftMax® Pro Software was used for the curve fit and calculations of IC50s.
AI-enabled image and data analysis
The second method was performed leveraging deeplearning segmentation within the Segmentation Is Not A Problem (SINAP) module. The Flexi-Protocol application was used to create a custom analysis protocol that extracted features across different colors and measurements for organoid objects, without identifying individual cells. Then, the Phenoglyphs™ module ran ML-based phenotypic classification using unsupervised and supervised methods to identify organoids based on their phenotypic characteristics, for example live/dead, or live/cytotoxic/cytostatic. A more detailed method description is presented in the Results section.
Results
Automated growth, monitoring, and treatment of healthy human intestinal organoids
A fully automated toxicity screening workflow for human intestinal organoids was developed using the CellXpress. ai system for machine learning-powered 3D culture and compound treatment, while the endpoint assay was performed using the ImageXpress HCS.ai system for AIpowered phenotypic analysis and high-content confocal imaging (Figure 1).
Figure 1. Workflow for drug screening in human intestinal organoids (HIOs). Organoids were seeded into 96-well U-bottom plates using the CellXpress.ai system, monitored by transmitted light imaging every 24 hours, treated with anti-cancer compounds on day 2. Live-cell imaging was performed on day 5 after seeding using the ImageXpress HCS.ai system with Hoechst, MitoTracker Orange, and Calcein AM staining. Endpoint analysis was performed with AI-driven phenotypic profiling using IN Carta software.
In this study, the CellXpress.ai system enabled automated organoid seeding in Matrigel, culture, including media exchanges that occurred every 48 hours, while transmitted light (TL) imaging was performed every 24 hours, enabling continuous, non-invasive monitoring of growth and morphology (Figure 3). Handling Matrigel and organoids is typically challenging, especially for process automation. Matrigel solidifies fast which, when pipetting, can cause domes to vary in size and organoid distribution— complicating endpoint analysis. The chilling platforms on the liquid handling deck allowed handling 50-80% Matrigel without solidifying, and optimized mixing and pipetting steps. This allowed user-defined settings of not only pipetting steps and volumes, but also the flow rate and mixing repetitions. In this protocol, we opted to use U-bottom 96-well plates that simplify organoid plating and imaging. Figure 2 shows the steps for organoid seeding, feeding, and monitoring.
The CellXpress.ai system enables accurate seeding of organoids starting from approximately 50 organoids per µL concentration, into 96-well U-bottom plates with 18 µL Matrigel domes per well. They were uniformly distributed with minimal variability in the initial organoid seeding (Figures 2 and 3). On the day of seeding, each well contained approximately 900–1000 small or immature organoid particles. Throughout the culture (5 days), the organoids self-organized, proliferated, and developed into characteristic spherical hollow structures, with a lumen surrounded by a layer of cells. By day 5 after seeding, each well contained around 75 fully grown organoids in control wells.
Figure 2. Automated seeding and monitoring of HIOs using the CellXpress.ai system. Organoids mixed with Matrigel were seeded into 96-well U-bottom Corning plates, 18 μL per well, and the Matrigel was allowed to polymerize as part of the seeding protocol. Then the media was added to the plates with a low flow rate. Plates were then moved to the incubator for 2 hours (incubation step), then the Feeding step performed imaging every 24 hours. Organoids were imaged every 24 hours using TL at 10X magnification, with the analysis protocol optimized for human intestinal organoids.
The next phase in the protocol was for compound addition from the master plate. Drug treatment included a three-phase automated protocol (Figure 4). First phase: Compound Addition applied pre-diluted compounds from the compound plate to the organoid plate, containing seven-point serial dilutions of eight compounds. Second phase: the organoid plate was incubated with compounds for 3 days using the Incubate step of the protocol. Third phase: staining of the organoids was done by adding 50 µL of staining solution per well (Figure 4 shows all three phases). In this third phase, organoids were stained for endpoint toxicity evaluation using the Feeding step in the CellXpress.ai system software. After that the washing step was performed using PBS (half of the volume was exchanged). The pipetting steps were set at a low speed of liquid flow (20 µL/sec), and the pipette tip was set at 4 µm above the bottom. Organoids were stained with a mix of three dyes: Hoechst (nuclei), MitoTracker Orange (mitochondria), and Calcein AM (viability). We used higher concentrations of dyes that are typically used for 2D culture (approximately 2X higher), also we allowed at least 1 hour for staining. During the staining plates were in the incubator
Figure 3. Monitoring on CellXpress.ai. Shown is an example of human intestinal organoids cultured in a 96-well U-bottom plate and imaged using transmitted-light (TL) microscopy. The panel displays TL images from all wells, each containing multiple organoids, acquired at 10X magnification across four imaging sites. CellXpress.ai enables continuous, non-invasive visualization of organoid morphology over time, allowing researchers to track growth, structural changes, or drug-induced effects without disrupting the culture.
Figure 4. This protocol from the CellXpress.ai system demonstrates a three-phase experimental workflow using HIOs in 96-well U-bottom Corning plates for drug screening. The protocol uses the Liquid-Following mode for precise compound addition. In the first phase, 50 µL of compound-containing medium, prepared at selected concentrations, is dispensed into each well. Then in the second phase, organoids were incubated with compounds for 72 hours. The third phase involves adding 50 µL of staining medium per well. This is followed by endpoint imaging to assess the phenotypic effects.
Phenotypic changes in organoids after compound treatment.
After staining, live imaging was conducted using the ImageXpress HCS.ai system with 10X magnification and confocal optics. Z-stacks of 18 optical sections at 8 µm intervals were captured to cover ~136 µm in depth, followed by 2D maximum projection for analysis (Figure 5). Live imaging is time sensitive and must be conducted immediately after staining. Organoids were then fixed and stained with Alexa488-Phalloidin to assess cytoskeletal integrity; however, for the data presented below, we used live imaging results.
Figure 5 illustrates the differential phenotypic responses of organoids to various chemotherapeutic agents, highlighting their distinct effects on organoid morphology. SN-38, Doxorubicin, and Staurosporine show strong cytotoxic responses, with organoids appearing structurally disrupted and non-viable. In contrast, Taxol and Etoposide preserve organoid morphology, suggesting a cytostatic effect where growth is inhibited but the organoid structures remain. Mitomycin C shows the combination of both effects.
Image analysis: Organoid segmentation and classification
Organoid image analysis was performed using IN Carta software, which offers advanced tools for image feature extraction, deep learning-based segmentation through the SINAP module, and machine learning-driven phenotypic classification via the Phenoglyphs module.
First, high-content image analysis was done using the Custom Module Editor (see Methods section). Cells were scored as positive or negative depending on the signal intensities for Calcein AM or MitoTracker (mitochondria). Cells that were scored positive for Calcein AM or mitochondria were defined as intact (live) or damaged (dead), respectively. Positive and negative cells per organoid were counted, and the average areas and average intensities of positive cells were measured. After analysis, concentration dependencies for different readouts were plotted as a 4-parametric curve fit (for 0.06–100 µM concentration range) to calculate IC50s for compound toxicity effects. SoftMax Pro software was used for the curve fitting and calculations of half maximal inhibitory concentrations.
Figure 5. This figure depicts representative maximum projection images captured on ImageXpress HCS.ai system using the standard confocal option (60 μm pinhole) at 10X magnification. The imaging used to assess phenotypic responses to drug treatments after 72-hour incubation with 5-fold serial dilutions across a 0–100 µM concentration range, performed in triplicate. Organoids were stained with Hoechst (blue), MitoTracker Orange (orange), and Calcein AM (green). Notable phenotypic changes: cytostatic effect (decreased size of organoids) for etoposide, mitomycin; or cytotoxic (organoids disintegrated, dead cells) for staurosporine, SN 38, and doxorubicin.
Figure 9 shows the concentration-dependency of the total live-cell areas for organoids defined as positive for Calcein AM staining. This approach allows for quantification of phenotypic changes and identifying effective concentrations of compounds using individual signal readouts. Using this approach, the analysis involves establishing linear intensity thresholds for organoids, nuclei, and cells across different strains. These thresholds may vary between experiments and can be influenced by compounds. For instance, certain compounds reduce the intensity of the Hoechst stain. The other limitation of this approach includes the requirement to consider various readouts using statistical tools to capture the complexity of biology.
Machine-learning algorithms allow a more unbiased and general approach for the development of an analysis protocol, which would automatically find organoids and score them as intact or damaged. Segmentation of objects, or the separation of objects from the background, is one of the most challenging steps in bioimage analysis, yet it gives researchers the ability to partition an image into biologically relevant structures such as organoids, nuclei, or cells.
Organoids were segmented using SINAP, a deep learning segmentation tool designed to find objects of interest and separate those from the background. SINAP is further refined with the Segment Anything Model (SAM) tool to enhance training and accuracy (Figure 6). The SAM tool allows a seamless and easy way to annotate organoid objects. Instead of tracing the line around organoids, which is laborious and less accurate, the SAM tool allows for selecting organoids by just clicking the cursor in the middle of the object. The program looks for similar pixels around the pointed region until it hits the object boundaries, allowing it to segment the object.
Figure 6. Image showing steps for deep learning segmentation models training: First image shows the step where it finds the organoids using the SINAP deep-learning tool for segmentation with the SAM tool, the red arrow points toward an organoid (green, 6.A) and the second image shows the mask generated by SINAP (pink, 6.B). The figure also shows the SINAP interface in the IN Carta software for analyzing drug-treated human intestinal organoids. This process includes several steps: 1. Images of organoids post-treatment are loaded into the main panel. 2. The user selects annotation tools on the right to mark individual organoids or regions of interest. 3. Annotated images are added to a training set, and steps 1–3 are repeated to build a robust dataset. 4. A deep-learning model is then trained using these annotations. 5. This trained model is applied to automatically segment and classify organoids across the full dataset.
SINAP was used to segment organoids from the fluorescent channel TRITC (Mitotracker). The model was initially trained on 20–50 manually annotated images, with iterative refinement through correction by the user. The SAM tool was used for rapid and accurate annotation.
After SINAP identified the objects, the multi-parametric analysis protocol from the IN Carta software was applied. Flexi-Protocol analysis extracted detailed morphological features, allowing for quantitative analysis of organoid response to various treatments.
The Flexi-Protocol extracted a comprehensive panel of 117 quantitative features per organoid. These included morphological descriptors (area, roundness, and perimeter), intensity metrics across three fluorescent channels (mean, maximum, and minimum intensity), textural features (entropy, granularity, and contrast), and spatial distribution patterns (radial intensity and heterogeneity). Analysis extracted features from DAPI, FITC, and TRITC channels.
This multidimensional feature set enabled high-content phenotypic profiling of organoid populations into distinct classes, based on the similarities of their measurements. Classification of organoids was carried out using the Phenoglyphs module, which combines unsupervised and supervised classification of organoids.
The Phenoglyphs module is a machine learning-based image analysis tool used for phenotypic classification of biological samples in high-content imaging workflows. It enables the identification and categorization of subtle morphological differences across cells, tissues, or organoids based on quantitative image features
The Phenoglyphs module enables advanced phenotypic classification of drug-treated human intestinal organoids. Following segmentation by SINAP, the Phenoglyphs module analyzed morphological features, such as size, shape, intensity, and texture, to detect strong or subtle structural changes induced by various treatments. The workflow begins with unsupervised clustering, grouping organoids based on phenotypic similarity without prior labels. This is followed by supervised learning, where users assign biological meaning to each cluster and train a model to classify organoids across the dataset. The trained model is then applied to large-scale image sets, allowing consistent, high-throughput identification of treatment effects. This combined approach enables precise discrimination of viable, stressed, and dead organoids, making The Phenoglyphs module a powerful tool for quantitative drug response profiling and phenotypic screening in intestinal organoid models.
Figure 7. A. Screenshot of the IN Carta software showing data review page and feature measurements results from the analysis. B. The Phenoglyphs module within the IN Carta software analyzing HIOs grown in a 96-well format. At the center, a confocal composite reveals organoids (60–250 µm) with the TRITC signal showing mitochondria (orange) and Calcein AM (green), blue DAPI-stained nuclei highlighting dense epithelial organization. The Phenoglyphs module extracts hundreds of features per organoid and clusters of organoids into classes with similar morphologies. Then we trained the module to classify organoids as intact or damaged.
During the analysis, first we applied unsupervised classification that grouped organoids into 20 phenotypic clusters, based on multiparametric fingerprints identified for each organoid. These were subsequently curated into two classes of organoid phenotypes: Intact organoids (larger, round, with high Calcein AM and Mitochondria signals), or Damaged organoids (smaller or disrupted, with lower viability or mitochondria signals). Subsequently, we conducted several iterative rounds of training to classify organoids into two defined categories. The training process included the manual reassignment of certain organoids to the appropriate category based on visual assessment of their phenotypes (Figure 8).
The final classification yielded percentage distributions for each phenotype class (e.g., 0.5% Live, 99.5% Damaged for SN-38), allowing concentration-dependency evaluation of compound effects. These phenotypic distributions were used to generate dose-response curves and calculate phenotype-based IC50 values (Figure 9), reflecting the concentration at which major morphological or viability changes occurred. Unlike single-parameter readouts, this classification approach uses a combined morphological and signal-based context, offering more robust phenotypic analysis.
Notably, when comparing concentration curves (Figure 9) and IC50s for machine-learning classification vs single measurement (Calcein AM), there were notable differences in measured toxicity effects for doxorubicin, etoposide, and mitomycin C (Table 1). The Positive Area Sum measures the total Calcein AM-positive area, reflecting the viable cells within organoids. While lower values indicate reduced viability, organoids can maintain structural integrity despite decreased Calcein AM signal at intermediate drug concentrations. Such differences were observed because of a marked decrease in Calcein AM intensity, while organoids still had an intact appearance at medium concentrations. Machine-learning analysis, which considered a large variety of measurements, presented the concentration-dependency that reflected an overall “damaged” phenotype, rather than just an intensity decreases.
Figure 8. The Phenoglyphs module was able to classify objects based on extracted features. After an initial unsupervised clustering, the user can refine phenotypic classes according to the experimental purpose, for example, Intact organoids (green masks) vs damaged organoids (red masks). After sequential rounds of refinement, the trained model can be applied to the whole dataset to categorize all organoids into phenotypic classes.
Figure 9. A. Concentration dependencies shown for the percentage of Live (intact) organoids, defined by machine learning, that reflect corresponding changes in phenotypes. B. Concentration dependencies shown for the Positive Area Sum that reflect decrease in number of Calcein AM positive cells and size of organoids caused by toxic compounds. Note a sharper decline of the curves on the right-hand graph
Furthermore, we were able to use machine learning to distinguish between the cytostatic or cytotoxic effects of compounds. Cytostatic effects, meaning mostly inhibition of cell proliferation, were defined as having the appearance of smaller but intact and live organoids indicating reduced growth rather than cell death. Cytotoxic effects, meaning effects causing cell death, were defined by the appearance of disrupted, fragmented, or dead organoids (Figure 10).
Machine-learning analysis using the Phenoglyphs module classified organoid phenotypes into three biologically relevant classes: Live, Cytotoxic, or Cytostatic (Figure 10). SN-38 and Staurosporine induced clear disruption of organoids; in contrast, Etoposide and Mitomycin C induced mostly cytostatic effects, except at the highest concentrations.
We demonstrate here an automated workflow for a toxicity evaluation assay using organoids and high-content imaging. In the above examples, we described approaches that can evaluate toxicity effects using AI-enabled phenotypic profiling. The use of the CellXpress.ai and ImageXpress HCS.ai systems provide a scalable solution for compound testing using organoids, while the AI-driven image analysis allows comprehensive phenotypic profiling relevant to phenotypic changes in the human intestinal organoid model.
Figure 10. A. The Phenoglyphs module was used to classify intestinal organoids into live, cytostatic, and cytotoxic categories. B, C. Representative examples for each class are shown for Live (green), Cytostatic (blue, pink), or Cytotoxic (red) effects, respectively. Live organoids appear bigger and intact, cytostatic are smaller but intact, and cytotoxic organoids are fragmented or disrupted.
Table 1. The table shows IC50 (µM) values for the drug’s effects. The % live organoids indicate the proportion of intact organoids in the well defined by machine learning classification. The Positive Area Sum measures the total area of positive (intact) live cells, which reflects cell viability. Calcein AMpositive area, reflecting viable cell number in organoids. Lower values indicate reduced viability, but organoids can appear intact despite decreased Calcein AM signal at intermediate drug doses. Machine learning captures this subtle damage better by considering multiple features beyond fluorescence intensity. The Positive Area Sum values reflected the areas of positive cells (for Calcein AM) in organoids, defined as a single readout. Low values for SN-38 and Staurosporine indicate high toxicity at the lowest concentrations of drugs, while higher values for Doxorubicin and Mitomycin C reflect lower toxicity effects. Notably, IC50s for Etoposide and Doxorubicin are much higher for ML-assessed classification, as seen in Figure 5.
Summary
This study highlights an AI-driven approach to toxicity screening using automated culture and compound testing on healthy human intestinal organoids. The integration of artificial intelligence enables accurate identification of toxic compound effects, with an automated screening process. By leveraging human intestinal organoids, researchers gain a more physiologically relevant and human-representative model compared to traditional animal-based methods. This workflow is well-suited for high-throughput screening due to its consistency and scalability. Overall, this AI-enabled platform delivers a reliable, efficient solution for toxicity assessment, enhancing both the precision and impact of scientific research.
References
- Weiß, T., Schmidt, R., Müller, L., & Becker, M. (2025). Matrix-free human 2D organoids recapitulate duodenal barrier and transport properties. BMC Biology, 23(1), Article 2. https://doi.org/10.1186/s12915-024-02105-7
- Miyoshi, J., Takahashi, Y., & Watanabe, M. (2021). Monolayer platform using human biopsy-derived duodenal organoids for pharmaceutical research. Scientific Reports, 11(1), 21147. https://doi.org/10.1038/s41598-021-00545-3
HUB Organoid Technology used herein was used under license from HUB Organoids.
To use HUB Organoid Technology for commercial purposes, please contact bd@huborganoids.nl for a commercial use license.