#!/perl/bioinfo: febrero 2024

26 de febrero de 2024

Cómo modelar proteínas con colabfold en tu GPU local

Hola,

hoy explicaré cómo he configurado ColabFold para ejecutarlo en hardware local, en concreto en una máquina con Ubuntu 20.04 que tiene una CPU Xeon CascadeLake Silver 4210R y una tarjeta gráfica NVIDIA RTX 3090. Puedes leer más sobre AlphaFold y ColabFold aquí o en este vídeo.

1) Necesité actualizar cuda, en concreto con la versión 11.8, algo que hice como se explica aquí:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt update
sudo apt install cuda-toolkit-11-8

2) Tras reinicar, actualicé la variable de ambiente $PATH añadiendo estas líneas a mi fichero .bashrc:

export PATH=/usr/local/cuda/bin:$PATH

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

3) Seguí las instrucciones para Linux en https://github.com/YoshitakaMo/localcolabfold?tab=readme-ov-file#for-linux . En mi caso tardó unos pocos minutos y sumó 15G al disco duro.

4) Probé que todo funciona con un fichero FASTA qee contiene varias secuencias, guardando los resultados en la carpeta multi/ :

colabfold_batch test.multi.faa multi/

Ahora resumo los resultados que obtuve:

Por defecto colabfold_batch se conecta a https://api.colabfold.com para hacer búsquedas de secuencias similares y construir alineamientos múltiples (MSA) en un formato similar a FASTA que se llama a3m. Por tanto esa parte del trabajo no se hace localmente y tendrás que usarla con medida. Si quieres saber qué versión de las bases de datos de secuencias de ColabFold estás usando puedes consultar https://github.com/sokrypton/ColabFold/wiki/MSA-Server-Database-History
Las primeras secuencias que usé para construir modelos en formato PDB tenían entre 114 y 162 resíduos y tardaban un par de minutos, pego aquí el log:
2024-02-26 13:05:56,639 Running colabfold 1.5.5 (d36504fad856a0e1df511c5b0434957707030319)
2024-02-26 13:05:56,862 Running on GPU
2024-02-26 13:05:57,354 Found 5 citations for tools or databases
2024-02-26 13:05:57,355 Query 1/29: test1 (length 114)
2024-02-26 13:05:58,348 Sleeping for 6s. Reason: PENDING
2024-02-26 13:06:05,308 Sleeping for 10s. Reason: RUNNING
2024-02-26 13:06:30,822 Padding length to 124
2024-02-26 13:06:58,791 alphafold2_ptm_model_1_seed_000 recycle=0 pLDDT=67.9 pTM=0.31
2024-02-26 13:07:00,321 alphafold2_ptm_model_1_seed_000 recycle=1 pLDDT=68.8 pTM=0.329 tol=9.09
2024-02-26 13:07:01,845 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=69.7 pTM=0.358 tol=2.28
2024-02-26 13:07:03,373 alphafold2_ptm_model_1_seed_000 recycle=3 pLDDT=69.8 pTM=0.367 tol=3.04
2024-02-26 13:07:03,374 alphafold2_ptm_model_1_seed_000 took 32.6s (3 recycles)
2024-02-26 13:07:04,871 alphafold2_ptm_model_2_seed_000 recycle=0 pLDDT=71.2 pTM=0.308
2024-02-26 13:07:06,323 alphafold2_ptm_model_2_seed_000 recycle=1 pLDDT=71.6 pTM=0.346 tol=2.14
2024-02-26 13:07:07,848 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=71.7 pTM=0.358 tol=2.38
2024-02-26 13:07:09,345 alphafold2_ptm_model_2_seed_000 recycle=3 pLDDT=71.8 pTM=0.365 tol=1.31
2024-02-26 13:07:09,346 alphafold2_ptm_model_2_seed_000 took 5.9s (3 recycles)
2024-02-26 13:07:10,984 alphafold2_ptm_model_3_seed_000 recycle=0 pLDDT=68.1 pTM=0.298
2024-02-26 13:07:12,529 alphafold2_ptm_model_3_seed_000 recycle=1 pLDDT=68.6 pTM=0.34 tol=4.11
2024-02-26 13:07:13,992 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=69.2 pTM=0.36 tol=2.49
2024-02-26 13:07:15,484 alphafold2_ptm_model_3_seed_000 recycle=3 pLDDT=68.8 pTM=0.367 tol=1.67
2024-02-26 13:07:15,485 alphafold2_ptm_model_3_seed_000 took 6.1s (3 recycles)
2024-02-26 13:07:16,987 alphafold2_ptm_model_4_seed_000 recycle=0 pLDDT=66.1 pTM=0.289
2024-02-26 13:07:18,435 alphafold2_ptm_model_4_seed_000 recycle=1 pLDDT=66.8 pTM=0.283 tol=5.61
2024-02-26 13:07:19,933 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=67.7 pTM=0.298 tol=1.03
2024-02-26 13:07:21,444 alphafold2_ptm_model_4_seed_000 recycle=3 pLDDT=67.9 pTM=0.318 tol=2.04
2024-02-26 13:07:21,445 alphafold2_ptm_model_4_seed_000 took 5.9s (3 recycles)
2024-02-26 13:07:22,931 alphafold2_ptm_model_5_seed_000 recycle=0 pLDDT=66.8 pTM=0.322
2024-02-26 13:07:24,403 alphafold2_ptm_model_5_seed_000 recycle=1 pLDDT=68.2 pTM=0.345 tol=9.46
2024-02-26 13:07:25,860 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=68.8 pTM=0.354 tol=2.3
2024-02-26 13:07:27,342 alphafold2_ptm_model_5_seed_000 recycle=3 pLDDT=69.4 pTM=0.358 tol=1.58
2024-02-26 13:07:27,342 alphafold2_ptm_model_5_seed_000 took 5.9s (3 recycles)
2024-02-26 13:07:27,369 reranking models by 'plddt' metric
2024-02-26 13:07:27,369 rank_001_alphafold2_ptm_model_2_seed_000 pLDDT=71.8 pTM=0.365
2024-02-26 13:07:27,369 rank_002_alphafold2_ptm_model_1_seed_000 pLDDT=69.8 pTM=0.367
2024-02-26 13:07:27,370 rank_003_alphafold2_ptm_model_5_seed_000 pLDDT=69.4 pTM=0.358
2024-02-26 13:07:27,370 rank_004_alphafold2_ptm_model_3_seed_000 pLDDT=68.8 pTM=0.367
2024-02-26 13:07:27,370 rank_005_alphafold2_ptm_model_4_seed_000 pLDDT=67.9 pTM=0.318
2024-02-26 13:07:28,679 Query 2/29: test2 (length 120)
2024-02-26 13:07:29,695 Sleeping for 9s. Reason: PENDING
2024-02-26 13:07:39,667 Sleeping for 9s. Reason: PENDING
2024-02-26 13:07:49,628 Sleeping for 6s. Reason: PENDING
2024-02-26 13:07:56,610 Sleeping for 6s. Reason: PENDING
2024-02-26 13:08:03,608 Sleeping for 5s. Reason: PENDING
2024-02-26 13:08:09,564 Sleeping for 6s. Reason: PENDING
2024-02-26 13:08:16,534 Sleeping for 7s. Reason: PENDING
2024-02-26 13:08:24,518 Sleeping for 5s. Reason: PENDING
2024-02-26 13:08:30,471 Sleeping for 7s. Reason: PENDING
2024-02-26 13:08:38,498 Sleeping for 5s. Reason: PENDING
2024-02-26 13:08:44,459 Sleeping for 6s. Reason: PENDING
2024-02-26 13:08:51,412 Sleeping for 9s. Reason: PENDING
2024-02-26 13:09:01,412 Sleeping for 9s. Reason: PENDING
2024-02-26 13:09:11,370 Sleeping for 8s. Reason: PENDING
2024-02-26 13:09:20,337 Sleeping for 8s. Reason: PENDING
2024-02-26 13:09:29,316 Sleeping for 6s. Reason: RUNNING
2024-02-26 13:09:39,703 Padding length to 124
2024-02-26 13:09:41,194 alphafold2_ptm_model_1_seed_000 recycle=0 pLDDT=73.9 pTM=0.55
2024-02-26 13:09:42,664 alphafold2_ptm_model_1_seed_000 recycle=1 pLDDT=73.8 pTM=0.549 tol=3.08
2024-02-26 13:09:44,110 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=73.6 pTM=0.549 tol=1.59
2024-02-26 13:09:45,593 alphafold2_ptm_model_1_seed_000 recycle=3 pLDDT=74.4 pTM=0.555 tol=1.67
2024-02-26 13:09:45,593 alphafold2_ptm_model_1_seed_000 took 5.9s (3 recycles)
2024-02-26 13:09:47,073 alphafold2_ptm_model_2_seed_000 recycle=0 pLDDT=76.7 pTM=0.565
2024-02-26 13:09:48,523 alphafold2_ptm_model_2_seed_000 recycle=1 pLDDT=77.1 pTM=0.57 tol=0.571
2024-02-26 13:09:49,977 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=76.7 pTM=0.569 tol=0.958
2024-02-26 13:09:51,421 alphafold2_ptm_model_2_seed_000 recycle=3 pLDDT=76.9 pTM=0.572 tol=0.881
2024-02-26 13:09:51,421 alphafold2_ptm_model_2_seed_000 took 5.8s (3 recycles)
2024-02-26 13:09:52,877 alphafold2_ptm_model_3_seed_000 recycle=0 pLDDT=75.6 pTM=0.542
2024-02-26 13:09:54,315 alphafold2_ptm_model_3_seed_000 recycle=1 pLDDT=75.9 pTM=0.548 tol=1.52
2024-02-26 13:09:55,763 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=75.9 pTM=0.552 tol=1.69
2024-02-26 13:09:57,218 alphafold2_ptm_model_3_seed_000 recycle=3 pLDDT=75.8 pTM=0.555 tol=0.883
2024-02-26 13:09:57,219 alphafold2_ptm_model_3_seed_000 took 5.8s (3 recycles)
2024-02-26 13:09:58,705 alphafold2_ptm_model_4_seed_000 recycle=0 pLDDT=73.9 pTM=0.56
2024-02-26 13:10:00,177 alphafold2_ptm_model_4_seed_000 recycle=1 pLDDT=75.1 pTM=0.57 tol=2.2
2024-02-26 13:10:01,620 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=75.4 pTM=0.571 tol=1.78
2024-02-26 13:10:03,076 alphafold2_ptm_model_4_seed_000 recycle=3 pLDDT=75.7 pTM=0.575 tol=2.04
2024-02-26 13:10:03,077 alphafold2_ptm_model_4_seed_000 took 5.8s (3 recycles)
2024-02-26 13:10:04,572 alphafold2_ptm_model_5_seed_000 recycle=0 pLDDT=75.2 pTM=0.573
2024-02-26 13:10:06,026 alphafold2_ptm_model_5_seed_000 recycle=1 pLDDT=76.2 pTM=0.585 tol=2.12
2024-02-26 13:10:07,498 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=76.2 pTM=0.587 tol=1.44
2024-02-26 13:10:08,958 alphafold2_ptm_model_5_seed_000 recycle=3 pLDDT=76.6 pTM=0.589 tol=1.21
2024-02-26 13:10:08,959 alphafold2_ptm_model_5_seed_000 took 5.9s (3 recycles)
2024-02-26 13:10:08,986 reranking models by 'plddt' metric
2024-02-26 13:10:08,987 rank_001_alphafold2_ptm_model_2_seed_000 pLDDT=76.9 pTM=0.572
2024-02-26 13:10:08,987 rank_002_alphafold2_ptm_model_5_seed_000 pLDDT=76.6 pTM=0.589
2024-02-26 13:10:08,987 rank_003_alphafold2_ptm_model_3_seed_000 pLDDT=75.8 pTM=0.555
2024-02-26 13:10:08,987 rank_004_alphafold2_ptm_model_4_seed_000 pLDDT=75.7 pTM=0.575
2024-02-26 13:10:08,987 rank_005_alphafold2_ptm_model_1_seed_000 pLDDT=74.4 pTM=0.555
2024-02-26 13:10:10,274 Query 3/29: test3 (length 162)
2024-02-26 13:10:11,241 Sleeping for 8s. Reason: PENDING
2024-02-26 13:10:20,230 Sleeping for 10s. Reason: PENDING
2024-02-26 13:10:31,195 Sleeping for 5s. Reason: RUNNING
2024-02-26 13:10:37,194 Sleeping for 6s. Reason: RUNNING
2024-02-26 13:10:44,153 Sleeping for 9s. Reason: RUNNING
2024-02-26 13:10:54,142 Sleeping for 10s. Reason: RUNNING
2024-02-26 13:11:05,109 Sleeping for 8s. Reason: RUNNING
2024-02-26 13:11:14,082 Sleeping for 6s. Reason: RUNNING
2024-02-26 13:11:21,030 Sleeping for 8s. Reason: RUNNING
2024-02-26 13:11:30,005 Sleeping for 9s. Reason: RUNNING
2024-02-26 13:11:39,984 Sleeping for 7s. Reason: RUNNING
2024-02-26 13:11:47,941 Sleeping for 10s. Reason: RUNNING
2024-02-26 13:11:58,903 Sleeping for 9s. Reason: RUNNING
2024-02-26 13:12:08,881 Sleeping for 5s. Reason: RUNNING
2024-02-26 13:12:14,891 Sleeping for 9s. Reason: RUNNING
2024-02-26 13:12:32,470 Padding length to 172
2024-02-26 13:13:00,100 alphafold2_ptm_model_1_seed_000 recycle=0 pLDDT=62.9 pTM=0.433
2024-02-26 13:13:02,186 alphafold2_ptm_model_1_seed_000 recycle=1 pLDDT=63.4 pTM=0.433 tol=8.27
2024-02-26 13:13:04,282 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=64.1 pTM=0.431 tol=8.02
2024-02-26 13:13:06,403 alphafold2_ptm_model_1_seed_000 recycle=3 pLDDT=63.8 pTM=0.427 tol=8.51
2024-02-26 13:13:06,404 alphafold2_ptm_model_1_seed_000 took 33.9s (3 recycles)
2024-02-26 13:13:08,535 alphafold2_ptm_model_2_seed_000 recycle=0 pLDDT=60.2 pTM=0.417
2024-02-26 13:13:10,637 alphafold2_ptm_model_2_seed_000 recycle=1 pLDDT=61 pTM=0.423 tol=6.09
2024-02-26 13:13:12,742 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=61.4 pTM=0.428 tol=3.33
2024-02-26 13:13:14,846 alphafold2_ptm_model_2_seed_000 recycle=3 pLDDT=61.2 pTM=0.425 tol=1.8
2024-02-26 13:13:14,846 alphafold2_ptm_model_2_seed_000 took 8.4s (3 recycles)
2024-02-26 13:13:16,979 alphafold2_ptm_model_3_seed_000 recycle=0 pLDDT=62 pTM=0.425
2024-02-26 13:13:19,099 alphafold2_ptm_model_3_seed_000 recycle=1 pLDDT=62.3 pTM=0.43 tol=7.21
2024-02-26 13:13:21,197 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=61.9 pTM=0.426 tol=4.32
2024-02-26 13:13:23,303 alphafold2_ptm_model_3_seed_000 recycle=3 pLDDT=62.1 pTM=0.427 tol=5.17
2024-02-26 13:13:23,304 alphafold2_ptm_model_3_seed_000 took 8.4s (3 recycles)
2024-02-26 13:13:25,461 alphafold2_ptm_model_4_seed_000 recycle=0 pLDDT=60.5 pTM=0.418
2024-02-26 13:13:27,552 alphafold2_ptm_model_4_seed_000 recycle=1 pLDDT=60.8 pTM=0.417 tol=9.52
2024-02-26 13:13:29,658 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=60.3 pTM=0.41 tol=9.23
2024-02-26 13:13:31,749 alphafold2_ptm_model_4_seed_000 recycle=3 pLDDT=60.5 pTM=0.411 tol=6.08
2024-02-26 13:13:31,750 alphafold2_ptm_model_4_seed_000 took 8.4s (3 recycles)
2024-02-26 13:13:33,905 alphafold2_ptm_model_5_seed_000 recycle=0 pLDDT=59.9 pTM=0.416
2024-02-26 13:13:36,038 alphafold2_ptm_model_5_seed_000 recycle=1 pLDDT=60.1 pTM=0.415 tol=9.96
2024-02-26 13:13:38,154 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=59.7 pTM=0.409 tol=3.89
2024-02-26 13:13:40,252 alphafold2_ptm_model_5_seed_000 recycle=3 pLDDT=59.4 pTM=0.415 tol=11.4
2024-02-26 13:13:40,253 alphafold2_ptm_model_5_seed_000 took 8.5s (3 recycles)
2024-02-26 13:13:40,294 reranking models by 'plddt' metric
2024-02-26 13:13:40,294 rank_001_alphafold2_ptm_model_1_seed_000 pLDDT=63.8 pTM=0.427
2024-02-26 13:13:40,294 rank_002_alphafold2_ptm_model_3_seed_000 pLDDT=62.1 pTM=0.427
2024-02-26 13:13:40,294 rank_003_alphafold2_ptm_model_2_seed_000 pLDDT=61.2 pTM=0.425
2024-02-26 13:13:40,295 rank_004_alphafold2_ptm_model_4_seed_000 pLDDT=60.5 pTM=0.411
2024-02-26 13:13:40,295 rank_005_alphafold2_ptm_model_5_seed_000 pLDDT=59.4 pTM=0.415
Como ves el propio script espera cuando el servidor remoto está ocupado.
Para cada secuencia problema obtienes figuras como éstas:

Hasta pronto,

Bruno

PD Cuando acabes de instalar deberías tener algo similar en tu fichero $HOME/.bashrc:

export PATH=/usr/local/cuda/bin:$PATH

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

export PATH="$HOME/colabfold/colabfold-conda/bin:$PATH"

Cuando no vayas a usar colabfold comenta estas líneas para usar perl y python del sistema

PD2 Me comentan colegas de ULiverpool que haciendo 800-900 MSAs al día en https://api.colabfold.com no han tenido problemas

PD3 Si no reconoce tu GPU mira posibles soluciones en https://github.com/YoshitakaMo/localcolabfold/issues/210

PD4: Puedes bloquear la versión de CUDA que hayas instalado con algo como:

sudo apt-mark hold cuda-toolkit-11-8

23 de febrero de 2024

lectura eficiente de ficheros comprimidos en el terminal

Al analizar datos genómicos a menudo tenemos que manejar ficheros de texto enormes, del orden de decenas de GB, que suelen estar comprimidos para no llenar el disco.

Imagen de https://the-chestnut.com

Por ejemplo:

-rw-r--r-- 1 contrera contrera 2.2G Feb 23 12:43 mergedpairs.tsv

comprimido con gzip ocupa casi la décima parte:

-rw-r--r-- 1 contrera contrera 255M Feb 23 12:43 mergedpairs.tsv.gz

A veces es necesario operar sobre este tipo de ficheros desde scripts o el terminal sin descomprimirlos en el disco. Por ejemplo, podemos contar el número de líneas con zcat:

time zcat mergedpairs.tsv.gz | wc -l
10205212

real    0m10.793s
user    0m10.689s
sys    0m1.726s

Otra opción un poco más rápida, qué descubrí en dos foros (1,2), es unpigz, que hay que instalar primero en Ubuntu:

sudo apt install pigz

time unpigz -dc mergedpairs.tsv.gz | wc -l
10205212

real    0m6.503s
user    0m10.742s
sys    0m3.391s

Otra opción que he encontrado es https://github.com/mxmlnkn/rapidgzip, que además tiene una interfaz python que permite indexar ficheros .gz, pero se puede usar en la línea de comando también:

python3 -m pip install rapidgzip

time rapidgzip -d -c -P 0 mergedpairs.tsv.gz | wc -l
10205212

real    0m2.833s
user    0m10.895s
sys    0m5.020s

Finalmente, he probado con https://github.com/facebook/zstd, que usa un formato de compresión menos estándar (.zst):

sudo apt install zstd
pzstd mergedpairs.tsv

time pzstd -d -c mergedpairs.tsv.zst | wc -l
mergedpairs.tsv.zst : 2273983394 bytes
10205212

real   0m1.600s
user   0m5.973s
sys   0m3.348s

Hasta pronto,

Bruno

7 de febrero de 2024

Browsing barley pangenes

Hi,

late last year we published a paper describing GET_PANGENES, a protocol to call pangenes, which are clusters of gene models/alleles found in genomic assemblies in a similar location. You can read all about it at https://doi.org/10.1186/s13059-023-03071-z . Using this approach you can produce figures like this, where you can see the pangene of interest in green:

Genomic context of barly pangene cluster HORVU.MOREX.r3.3HG0311160 (green arrows), which corresponds to barley locus HvOS2. Figure from https://doi.org/10.1186/s13059-023-03071-z

As we do research on barley breeding and adaptation, we thought it would be useful for us and others out there to have way of inspecting barley pangenes, for instance to check whether a gene of interest is conserved or polymorphic across the barleys sampled by in the pangenome (n=20) put together by Jayakodi et al.

This exactly what you can do, at the protein sequence level, at https://eead-csic-compbio.github.io/barley_pangenes , where you can scroll pangenes along chromosomes, with MorexV3 positions; genes not found in MorexV3 lack a position therefore and are shown with a hash (#):

You will notice that pangenes with occupancy > 1, ie containing gene models found it at least two barleys, can be clicked to display a multiple protein alignment with help from the NCBI msaviewer:

There you can easily zoom in to regions of interest and print or export the alignment in FASTA, PDF or SVG format (high quality).

Hope this can be useful to the barley genomics community,

Bruno