![\"\"](\"https://substackcdn.com/image/fetch/$s_!2aJw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F200aeb67-5980-497b-8f31-e99c3683090c_640x155.png\")
I first read about Lens.org via a tweet on my way back from a conference in April 2018. There seemed to be something in the water at the time, as they was an explosion of new discovery services and idexes in the past few months, including Digital science's Dimensions, 1Science's 1Findr, Scilit and the new resurgent challenge to Google Scholar posed by Microsoft Academic.
At the time, I took a quick look at Len's about page to get a better sense of what the data included.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!sXOq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9c38f5c-67df-4bd6-a607-4ccd4afa905a_640x414.png\")
I immediately noted that Lens had a strong focus on patent searching (it's predecessor Patent Lens was built in 1999 to \"render the global patent system more transparent\"), and article search was the area I was most interested in so I perhaps prematurely dismissed Lens as mainly Microsoft Academic Graph (MAG) + patent searching.
However I have come to realise that Lens might in fact be a far more exciting development than I thought.
While it is true that the scholarly search portion of Lens might be perhaps mostly dominated by the voluminous data from Microsoft Academic Graph (MAG), Lens is far more than the sum of it's parts by combining open data from half a dozen open data sources.
The other significant thing about Lens that differentiates it from the other search discovery and citation indexes is that it is run by Cambia a non-profit that seems committed to produce a open, free to use alternative to commerically owned and licensed indexes.
Most importantly, Lens is not just open data but there is a surprisingly feature rich interface that goes with it that can go toe to toe to many commerical interfaces. Given that the tool is still very new (Scholarly citations from Microsoft Academic Graph was added only 2 weeks ago) , this is pretty impressive and the speed of updates so far, makes me wonder how far it will go, though of course it is still early days.
This is going to be a detailed review in two parts. The first part will focus on the features offered by Lens, wth a focus on the Scholarly search portion of Lens.
This will be followed by a simple test of the data in Lens by extracting items with a given affiliation and comparing the overlap with data from Scopus/Scival.
A detailed analysis of the results will be provided in my next post.
Table of content
Lens is supported by Queensland University Technology and Cambia, which is self-described as \"a non-profit social enterprise that for over two decades has been dedicated to democratizing problem solving using science and technology.\"
I have never heard of Cambia until now, but I understand they have been in the patent search space (starting from life sciences before expanding to other patents) since the 90s. With funding from organizations like Rockefeller Foundation , Bill & Melinda Gates Foundation they have slowly expanded the capabilities of Patent Lens. pooling data from various Patent offices around the world, and added innovative features like Patent Sequence Search . I'm probably not doing full justice to the history of Cambia and Lens, please read the timeline for more details.
Lens makes the following pledge.
.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!FQxF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e8fa5e5-488e-47ea-8029-63919dc607f1_271x320.png\")
Open, free, privacy - pledges by Lens
Privacy conscious, Free (no premium options) and Data that is open ? Sounds too good to be true. The librarian in me wonders about sustainability and business models but as this post focus on the features of Lens, I shall refrain from commenting further.
The most interesting aspect of Lens to me besides the patent related functionality, is how it blends together data from various open data sources - including
This is very similar in concept to OpenAIRE's DOIBoost project which aims to enhance Crossref metatdata with MAG, ORCID and Unpaywall data but besides the difference in sources used, Lens has a full infrastructure around it and allows anyone to query and use without any technical expertise for free.
The idea here is that by combining all these open sources, one can cover for the gaps in any single data source to provide a super \"meta\" record that is better than any individual source.
Below shows one particularly complete record in the search results.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!YPR2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7aaeda92-be8a-4b9a-a80e-7ef097aff3aa_640x354.png\")
A particularly complete record in Lens search engine result page
Besides Unpaywall's links to open copies, the latest version of Lens even tries to use the OCLC registry to try to redirect you to your institution's resolver to access items via institutional subscriptions.
Unfortunately, my hunch is not that many libraries maintain their Openurl settings in WorldCat registry so this might not work for many people. Also the release notes that this feature currently works only for records with doi, though this might change in future versions.
Most of the tags listed such as \"open access\", \"patents\", \"affiliation\", \"Field of study\" , \"Clinical trial\" reflect data from various sources, I can hazard a guess where most of them come from.
Still, in case you are wondering, the \"Full text\" tag also indicates Lens has the full text of this paper indexed probably via CORE possibly the largest open repository of full text of Scholarly item.
Clicking into the record, you will see this impressive full record.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!LBi8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe42e4659-5552-4a48-972e-015d71e33e8a_640x510.png\")
A particularly complete Lens record - Full record view
On the top right you can see various pid (permanent IDs) from Crossref, Microsoft Academic, Pubmed, Pubmed Central and a pid created by Len itself, so you can see what sources the combined record uses & click through to resolve the pids.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!EySq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a612089-4bad-4f9f-98ad-f2466454222c_640x268.png\")
PIDs in the Lens record
To learn more about the Lens ID, refer to this help file , but it is essentially \"15 digits long, broken in five groups where every 3rd digit is separated from the next with a dash (-)m followed by a check digit.
Below that, you get various information, in particular citation counts from Scholarly material and patents, references in the paper, funding info, links to Open access version and more.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!KCYo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e07c855-342c-4396-877a-1c3cc24d54fc_300x640.png\")
Citation metrics in Lens full record
In many ways this combined record compares well against commerical A&I records from Scopus, Web of Science and Dimensions.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!54Ya!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5dbdfa5e-9e67-4048-9e0a-fb7cea1c971d_640x581.png\")
Lens Full record is comparable to commerical alternatives
Of course, not all the records will be that complete, it all hinges on the completeness of the various sources Lens drawn from.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!CXdp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68787ee2-ea9b-42af-992e-72b5e04623f2_640x212.png\")
Lens default search box offers Patent (default) and article search
On the Lens homepage you can choose to search for patents or by articles (clicking on the \"Scholar\" button). If you just enter your query and hit enter, patents will be the default search.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!-zs1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb87b768e-1278-4863-b52a-c66819e9c240_640x228.png\")
Use the drop down arrow to do a structured (advanced field) search
Perhaps of interest to power users will be the \"Structured Search\" option, this is effectively the advanced search. The \"classification viewer\" option is for patent search.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!8GWg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58d45c4d-a1a6-4045-99f5-cfd9b76b16f0_640x406.png\")
Lens - structured field search for articles
The structured search will have a toggle to switch between Scholarly articles and patents.
Looking at the fields for Scholarly search can be awe inspiring, as you can see there is a wide variety of possible fields to search through , all thanks to the various data sources Lens pulls from.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!bkIJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa8294c3-95d3-4fa5-8fba-0e6eb0e141bf_458x640.png\")
Some fields you can use in structured field search
You get the usual title, abstract, author, keywords, pub type etc fields
Significantly you get \"Author affiliation Name\", so you can search by institutions, something you can't do easily with most free services (the exception is Microsoft Academic though you can't filter direct but need to requite on the semantic search - type and select from drop down) . I'll talk more about this later.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!eRQN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9ca1f8-b126-449c-aff6-fc0f80bf23a4_320x196.png\")
Restricting to affiliations in Microsoft Academic
You also get less commonly seen fields related to
Controlled vocabulory fields like MESH (from Pubmed), Field of Study (from Microsoft Academic)
Funding (eg Funding name, Funding Country, Funding ID etc)
Clinical trials (eg Clincial trial registry, Clincal trial ID)
Chemical substances (e.g Chemical Registry Name)
Reference (by Reference ID and Reference by Len Id)
Sources (e.g Source Country, Publisher, ISSN, Source subject etc)
Conference (e.g name, location, year etc)
Open Access (e.g Color, license)
I'm not too familar with some of these fields, but it's obvious some like the Open Access data is drawn from Unpaywall , a lot of STEM related fields are drawn from Pubmed, sources are probably from Crossref etc, while the more common ones are merged from more than one source.
In a way, the fact that it has a lot of filters is also a weakness. It isn't easy to tell how completely populated the fields are. For example, where is the Funding data drawn from? How complete are they? Does every record have a author affiliation field? That said there are flags you can use to restrict your search.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!PSDi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83a2c428-06f5-4593-a30e-6d006444d965_197x320.png\")
Flags in Lens
For example, you could restrict your search to only records with funding data, if they were open access, or even only if Lens had full text indexed.
Lens has full blown support for boolean operators which together with comprehensive filters, search history , saved queries and saved collections (compares to \"saved lists\" features for many databases) means Lens is feature rich enough even for power users. But in some sense this may not be as relevant as it seems as I wonder how many users will use Lens as a purely discovery tool when Google Scholar exists.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!z8nv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d16ecc-0d55-4c1f-8e42-2acb86a312ec_640x244.png\")
Search History in Lens
I would rate the emergence of Google Scholar as a scholarly discovery tool has perhaps the most significant development in this field so far in the last 20 years. While it is by no means perfect and lacks some features wanted by power users, it's existence and popularity is the reason why it's extremely difficult now to charge for a general purpose scholarly article discovery index. (My view is Scopus and Web of Science mainly persist due to their importance as citation indexes, while other subject specific A&I have specific disciplinary features).
This is reflected perhaps by the fact that one of the newest entries - Digital Science's Dimension provides a freemium discovery search that matches up with Google Scholar but adds the conventional filters that Google Scholar lacks and even throws in some visualizations of results.
Microsoft Academic is almost eeriely similar as a discovery tool. (To be fair both Dimensions and Microsoft Academic boast about using latest Machine learning and Natural language processing techniques for entity recognition and topic modelling, but we are talking about features users can use and it's arguably how much of what techniques actually lead to an improvement in relevancy of results compared to what Google Scholar is doing)So it is no surprise that Lens as a Scholarly discovery tool follows the same playbook when displaying the results and shows not only results, filters but also a visualization panel on the right.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!N2Ua!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc173397e-dafa-4a13-bb3d-a4a1684894e5_640x370.png\")
Search result page in lens with filters and visualization panel
Still Lens is perhaps servicable as a discovery service if for some reason you want a change of pace from Google Scholar and use a tool that is designed for power users.
For sure the size of the index is not an issue, as of writing Lens has over 194 million records , which would cover the vast majority of peer reviewed material and Scholarly items you would want to find.
For comparison as at time of writing Microsoft Academic is boasting of 200 million records - which makes me wonder again about how much this one source is impacting Lens and whether the Scholarly search of Lens is just Microsoft Academic really.
That said searching Microsoft Academic and Lens with the same search keywords will give you very different results (far fewer in Microsoft Academic) because Microsoft Academic uses a \"semantic search\" rather than traditional keyword matching!
For further context on sizes of databases, Latest research as of 2018 estimates the sizes of the following databases
Google Scholar - 389 million
Proquest (selected 19 databases) - 279 million
Ebscohost (Sslected 375 database) - 132 million
Bielefeld Academic Search Engine (BASE) - 117 million
Web of Science (10 databases including core collection) - 105 million
Scopus - 72 million
Web of science (core collection) - 67 million
So clearly Lens is up there with the biggest , except for the might Google Scholar of course.
You could use Lens as a discovery tool by typing keywords but the fact that the default ordering is by Scholarly cites gives you a slight cue perhaps that this isn't the main purpose?
Still you can change the sorting to any of these
Relevancy
Scholarly citations
Patient citations
Publication date
Source Subject
Source type
Somewhat similar to Dimensions, you can see visualizations of the results set on the right of the screen. You can choose to open this as a full screen.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!uhFi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa269090c-f6d6-4868-ba2d-df47b76d6b11_640x492.png\")
Full page visualization panel in Lens
By default you get institutions displayed as a logo grid as well as various visualizations of authors, journals, Field of Study, year of publication etc.
But you can visualize almost any of the other fields such as Funding, Open Access Color, Open Access License, Conference related fields, Citation type etc.
I was wondering what the \"citation type\" visualization does, but it basically shows you the source where the data comes from.
For instance take this visualization of the research set by searching for items with author affiliation for my institution (filtered from 2013-2017).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!nasa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff71d65b3-43bc-4b60-82b7-7b9005858224_640x640.png\")
Citation type in Lens
As you can see, the result set has 2,345 records but magid (Microsoft Academic ID) alone contributes 2,329 records. Crossref (dois) are almost as numerous and there are very little from Pubmed (pmid) or Pubmed Central (pmcid) sources.
This makes a lot of sense since my institution does not have do research in life sciences. It also shows there is a huge overlap between Crossref records and MAG and in this case at least MAG \"knows of\" almost all the records in the result set, though no doubt the merger of both Crossref and MAG might provide better quality of data than just using MAG.
Of course, the real selling point about Lens is that it shows you the citation counts of result sets at various levels up to institutions.
Let's talk about that.
In many ways, tools like Scopus and Web of Science are valued today less for discovery per se but the citation metrics they provide. I suspect many libraries subscribe to them for this very reason.
To be fair Google also has metrics and you can find papers citing papers you are looking at but functionality is limited, though admittedly the author profiles in Google Scholar have had good takeup rates.
What Google Scholar and most freemium services like Dimensions lack is the ability to compare by institutions and to export in large batches.
Many institutions pay Elsevier and Clarivate large sums for Scopus/Scival and/or Web of Science/Incites for access to such features for benchmarking of institutions. Research Offices and libraries hunger for information on such benchmarks and measuring of impact .....
Can Len do some of this for free? Surprisingly yes.
Like any normal citation index, Lens allows you to find Scholarly citations of each paper by clicking on it. Until two weeks ago, this citation count included only open Crossref citations which is known to be lacking due to hold outs like Elsever, IEEE, ACS, but now it includes citations from Microsoft Academic Graph, which of course makes Lens citations far more complete, thiough perhaps at the cost of some accuracy as MAG data is mostly automatically extracted from bots.
Another piece of good news. Lens allows you to filter by affiliation, a feature usually not available in free versions. To do so, go to the structured search, select Scholarly toggle and use the author affiliation field.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!B4xN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b77595c-6349-4a9f-97e1-8b9584b0103c_640x423.png\")
Using structured search to search for items with affiliation
For my institution this gets me almost 6k records, you can add a bunch of filters after that, but the main trick is you can do the same visualization as before to study aspects of the outpost of your institution.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!fDWo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd113629-a5d6-409c-ab24-65ec5cfa1f7f_640x379.png\")
Visualization of institution output in Lens
For example you get handy stats for total cites from Scholarly material and patents to the almost 6k publications of my institutution. So for my set of results of 5,933 records
134 them were cited by patents
This came from 382 patents
There were 496 total patent citations
3,962 records were cited by Scholarly items
This came from 75,226 Scholarly items
With total cites of 90,855
Cites from patents rather than scholarly items is perhaps a different way of measuring impact of articles than the usual citations. Elsevier's Scival has a tab showing \"Economic impact\" which shows this very metric. I will talk more about this in the next blog post.
Besides citation metrics, you can also get the same visualizations as before, in particular you can see how much of the output is open access, by license, by top journal titles, collobrating institutions etc.
But besides showing useful statistics, Lens would be less useful as a tool, if one was unable to batch export the results for further study.
In fact, Lens is very capable here too. Once you have registered an account (possibly linked to Linkedin or ORCID) you can add all the results in a search and add to a collection.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!5BMU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b2e5816-ee52-44a8-9596-b4a0fbfecf59_640x428.png\")
Adding results to collections in Lens
Below shows some collections I have created. Some are for institutions for my comparisons and some are for authors.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!-nis!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1054f36-6b98-45e5-9345-ca7231bbed8a_640x331.png\")
My collections in Lens
Such collections can be set up to be shared publicly or privately and most importantly do a batch export up to 50k records in CSV, BibTeX, RIS, JSON formats
![\"\"](\"https://substackcdn.com/image/fetch/$s_!g7YR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30b4bf2a-dfbe-4f70-a1fe-a5f8e89cc6b2_640x462.png\")
Exporting up to 50k records in Lens
The last is a particularly generous feature, given that Scopus only allows 20k exports. This should be sufficient for most bibliometric analysis of even the largest institutions (in a batch or two).
It is somewhat disappointing though that while you can export a bunch of fields , unlike Scopus or Web of Science you can't export the references, which can be useful for bibliometric analysis using tools like VOSviewer.
One workaround which doesn't quite scale for large datasets is to go to the full record of every item, click on the citations tab.
Then lower down, select \"XXX references\" and click on view articles in Scholar Search
![\"\"](\"https://substackcdn.com/image/fetch/$s_!Jmh0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03210a2d-3547-45ef-ae1c-e3cae0016f6f_320x198.png\")
On the resulting search result page, add to a new collection. Rinse and repeat as required, then do a batch export. This works also if you selected \"XXX Scholarly Citations\" instead.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!j3AK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a10ff92-8b3e-4337-b256-a8ea1a1727ba_320x190.png\")
Here's a tip, you can search directly for references of a an item in Lens by going to the structured search , choosing the field \"Reference ID\" and entering the Lens ID of the item (without hyphens).
Lens is clearly still a state of evolution, so what follows is just a preliminary assessment
Compared to expensive tools like Scival and Incites or Dimensions (for analytics), it lacks more sophisticated metrics like Relative Citation Ratio (RCR) , Field Weighted Citation (FWCI) and even the simple H-index etc though if you really wanted to, you can sort by citation count and manually get the H-index. I've found sorting by citations somewhat quirky and occasionally the sorting doesn't work properly, though reloading often works.
Lens currently still lacks an API but I'm told this will be coming soon.
Of course, Scival and Incites have much better features for benchmarking than Lens (e.g. the function of plotting various institutions against each other), but to be fair Lens should be matching up with Scopus/Web of Science rather than Scival/Incites and against them Lens is more than capable function wise.
That said, citation indexes need to be trusted to be accepted. As of now, there are only 2 citation indexes that are taken seriously - Scopus and Web of Science. Both of them are over a decade old and are used if (only partly) for many real world decisions in particularly University rankings and Faculty assessment.
Google Scholar is the third one often mentioned by faculty who like the higher citation counts, but IMHO is unlikely to be seriously accepted in say University rankings because the data is not transparent and cannot be easily studied (lack of infomation, batch exports and API) .
Digital Science's Dimensions is my bet to be the third widely accepted citation index (because they have made in roads by working with institutions) , though it will take years if not decades for this to occur.
But none of these are free, will Lens make the grade as a accepted citation index eventually?
The elephant in the room of course is I have been talking about features of Lens, without considering the quality of the data. Given that Microsoft Academic Graph is a big part of the data, one might suspect the quality of the data relies heavily on that source. One also wonders at how well the merging process occurs, for example which source takes priority if there is a conflict? How does the mapping occur between publication types listed by Crossref, PubMed and Microsoft Academic Graph?
I will cover in greater depth a couple of tests I did in my next blog post, but here are some preliminary results I got in a test I did by comparing items in Lens with author affiliation - \"Singapore Management University\" against the output I obtained from Scival.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!Z9up!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071af46d-67d4-4730-8558-1f8d995f5761_549x411.jpeg\")
Output was matched first using DOI , followed by a fuzzy match of titles ( matched only if pub year is less than or equal to a year).
As you can see the overlap is fairly respectable (52.5%). A slightly surprise was that the output of Lens for my institution (2,345) was only slightly bigger than Scival's (2,271).
A interesting study to compare against would be this study by Leiden University on \"Accuracy of affiliation information in Microsoft Academic\", though the study used Web of Science to compare not Scopus, given that Microsoft data is a big component of Lens,
I then analysed the differences in results, with led to quite interesting insights but that will have to wait for the next post.
OLMo 2 1B is a new generation of open-source language models launched by the Allen Institute for AI, with a parameter scale of about 1 billion. It provides excellent natural language understanding and generation capabilities while maintaining a small size, and is suitable for a variety of downstream tasks.
In this post, we will give a run-down of the complete installation process, fine-tuning, and evaluation of the OLMo-2-0425-1B model from AllenAI. We explore how to use LoRA (Low-Rank Adaptation) to fine-tune the model efficiently and evaluate its sentiment classification ability with a custom dataset, all running locally with MPS acceleration.
OLMo-2-0425-1B is a 1.1 billion parameter decoder-only language model released by AllenAI on May 1,2025. As part of the OLMo series, it is trained from scratch using high-quality, open-source datasets with a strong emphasis on transparency, reproducibility, and research accessibility.
Key characteristics of the model include:
OLMo-2-0425-1B is not instruction-tuned or chat-optimised by default. Instead, it is designed as a research-friendly base model\u2014ideal for experimenting with new downstream tasks such as classification, summarisation, or dialogue systems. So it is easily tested on a local computer because:
conda create -n olmo-fresh python=3.10 -y\nconda activate olmo-fresh\npip install torch torchvision torchaudio\npip install transformers datasets peft gradio\nAnd if you keep meeting the problem like this
RuntimeError: operator torchvision::nms does not exist
or
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
It means you may have manually installed torch / torchvision from different sources or multiple libraries loaded different versions of the OpenMP (libomp) dynamic library, causing conflicts. We recommend that you uninstall torch/torchvision or create a new virtual environment. This is the quickest way to solve these kinds of problems.
OLMo 2 1B is supported in transformers v4.48 or higher:
\npip install transformers>=4.48
You can use OLMo with the standard HuggingFace transformers library:
\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai/OLMo-2-0425-1B\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai/OLMo-2-0425-1B\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\nAfter running the script above you will see the output like this:
\nLanguage modeling is a type of language learning methodology that is concerned with acquiring linguistic competence without having the necessary raw input in the target language (L2).
\nNatural language processing (NLP) is the study of techniques for computers to process, understand, and manipulate human languages. Natural language processing is a broad discipline that also focuses on aspects of language acquisition.
\nTranslation models is a type of software program used in translation to translate text from one language to another. The most popular translation model is statistical
\n`
As we mentioned before OLMo-2-0425-1B is designed as a research-friendly base model, so we want to test a new task Sentiment Classification on OLMo-2-0425-1B. We want to classify the sentiment contained in a given sentence as positive or negative. Here is an example
\nInput sentence:
\n\"I really hated the service at this restaurant.\"
Output:
\nNegative
Here is the source code : Git repository
\nFine-tuning is the process of adapting a pretrained language model to a specific downstream task\u2014such as sentiment classification\u2014by continuing its training on a smaller, task-specific dataset. Instead of training a model from scratch (which is computationally expensive and data-hungry), fine-tuning allows us to leverage the general linguistic knowledge already captured by large-scale pretrained models.
\nHowever, full fine-tuning of large language models (LLMs) like OLMo-2-1B can be resource-intensive, especially when running on consumer hardware.
\nLoRA (Low-Rank Adaptation of Large Language Models) offers a solution to this challenge. It is a parameter-efficient fine-tuning technique that:
\nIn our case, LoRA allowed us to fine-tune OLMo 2 1B on a MacBook Pro using MPS acceleration without running out of memory.
\nHere's a simplified visual of how LoRA operates within a transformer layer:
\nW (frozen)\n + A @ B \u2190 (trainable low-rank adapters)\n ---------\n \u2193\n Modified Attention Output\nWhere:
\nWe use GPT-4o to generate a small synthetic sentiment classification dataset, for each sentence we give it a label, here is the one example data in the dataset:
\n{\n \"text\": \"I absolutely loved the product.\",\n \"label\": \"positive\"\n}\ntrain.json(160 data) and val.json (40 data)for training and validation.We used PEFT to apply LoRA to the OLMo model and fine-tuned it on our sentiment dataset. Here is my full project path:
\nOlmo/\n\u251c\u2500\u2500 data/\n\u2502 \u251c\u2500\u2500 full.json # full dataset\n\u2502 \u251c\u2500\u2500 train.json # training subset\n\u2502 \u251c\u2500\u2500 val.json # validation subset\n\u251c\u2500\u2500 train.py # main LoRA fine-tuning script\n\u251c\u2500\u2500 infer.py # inference script\n\u251c\u2500\u2500 test.py # accuracy evaluation\n\u251c\u2500\u2500 train_loss_log.json # training loss record (JSON)\n\u251c\u2500\u2500 split_data.py # utility to split full.json\n\u251c\u2500\u2500 visualize.py # loss plotting script\n\u251c\u2500\u2500 olmo-lora-adapter/ # saved LoRA adapter\n\u251c\u2500\u2500 checkpoints/ # chech point save dir\npeft_config = LoraConfig(\n r=8,\n lora_alpha=16,\n task_type=TaskType.CAUSAL_LM,\n lora_dropout=0.1,\n bias=\"none\",\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]\n)\ntraining_args = TrainingArguments(\n output_dir=\"./checkpoints\",\n per_device_train_batch_size=1,\n per_device_eval_batch_size=1,\n eval_strategy=\"epoch\",\n save_strategy=\"epoch\",\n num_train_epochs=10,\n learning_rate=2e-4,\n logging_steps=10,\n report_to=\"none\",\n fp16=False\n)\nWe also defined a LossLoggerCallback to track and save loss during training:
[\n {\"step\": 10, \"loss\": 12.3},\n {\"step\": 20, \"loss\": 9.1},\n ...\n]\nSaved as train_loss_log.json. And here is the loss curve
\n![\"Train](\"https://github.com/CHeN-Ya0/Images/blob/main/loss_curve.png?raw=true\")
We defined an infer.py script to evaluate single sentences:
prompt = f\"<|user|>\\nClassify the sentiment of this sentence: \\\"{sentence}\\\"\\n<|assistant|>\\n\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(device)\noutputs = model.generate(**inputs, max_new_tokens=5)\nPredicted label is extracted from the generated output.Here is one example:
\n![\"Infer](\"https://github.com/CHeN-Ya0/Images/blob/main/infer.png?raw=true\")
An evaluation script was used to compute classification accuracy on the val.json:
correct = 0\nfor example in val_set:\n predicted = classify(example[\"text\"])\n if predicted == example[\"label\"]:\n correct += 1\naccuracy = correct / len(val_set)\nWe find the accuracy is 1 and here is the confusion matrix:
\n![\"Train](\"https://github.com/CHeN-Ya0/Images/blob/main/confusion_matrix_eval.png?raw=true\")
\nThe model was able to correctly classify a large portion of examples after just 10 epochs.
Through this experiment, we tested the real-world usability of OLMo-2-0425-1B, a 1.1B-parameter language model developed by AllenAI, in a local fine-tuning and inference setting. Rather than evaluating it solely as a pretrained base model, we explored how well it could adapt to a specific downstream task\u2014sentiment classification\u2014under resource-limited conditions.
\nOur results show that OLMo is a practical and flexible foundation model. With the help of parameter-efficient fine-tuning (LoRA), we were able to achieve reasonable performance using a custom dataset, all on a MacBook Pro without GPU. The model converged quickly, produced intelligible outputs, and showed a steady improvement in validation loss and classification accuracy over multiple epochs.
\nHowever, the testing also revealed some limitations. While OLMo handles structured prompts well, its output format can sometimes drift without proper formatting or prompt engineering. Additionally, compared to larger instruction-tuned models, it may require more task-specific guidance to maintain robustness in edge cases.
\nOverall, OLMo-2-0425-1B proved to be both efficient and adaptable, making it a strong candidate for researchers or developers looking to experiment with open models locally. Our testing confirms that even without cloud-scale hardware, OLMo can deliver meaningful results when paired with the right tools and training strategies.
","doi":"https://doi.org/10.59350/n8ant-1630","guid":"https://doi.org/10.59350/n8ant-1630","image":"https://hub.researchgraph.org/content/images/2025/05/olmo-logo-2.jpg","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1747267200,"rid":"1nsx6-ztq20","summary":"1 Overview OLMo 2 1B is a new generation of open-source language models launched by the Allen Institute for AI, with a parameter scale of about 1 billion. It provides excellent natural language understanding and generation capabilities while maintaining a small size, and is suitable for a variety of downstream tasks.","tags":["Artificial Intelligence","Toc"],"title":"From Zero to Classifier: Building a Lightweight LLM with OLMo 2 on Your Laptop","updated_at":1788597778,"url":"https://hub.researchgraph.org/from-zero-to-classifier-building-a-lightweight-llm-with-olmo-2-on-your-laptop/","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"Reposted from Paired Ends at\u00a0https://blog.stephenturner.us/p/llm-translate-documentation.
---
Most of the developer tooling for AI/LLM training and evaluation is Python-centric, but just over the past few months we've seen a surge of new tooling for AI/LLM\u00a0applications\u00a0for the R ecosystem.
ollamar\u00a0and\u00a0rollama\u00a0provide wrappers around the Ollama API allowing you to run LLMs locally on your machine. I recently wrote a few posts, one demonstrating how to use ollamar, and another demonstrating a package that uses ollama internally.
elmer\u00a0is a new package in the tidyverse that allows you to interact with many different LLMs via R (Claude, ChatGPT, Gemini, and Ollama too.
mall\u00a0is an interesting one that provides an easy way to run multiple LLM predictions against a data frame (sentiment analysis, summarization, classification, extraction, translation, etc).
pal\u00a0provides LLM assistants for doing things like highlighting code and asking for things like roxygen documentation, testthat tests, etc.
Shiny Assistant\u00a0helps you explain how things work in Shiny, and can help build Shiny apps for you (in either R or Python).
The\u00a0lang\u00a0package (source,\u00a0documentation) is an interesting new addition to the\u00a0mlverse\u00a0in R. From the documentation:
lang\u00a0overrides the\u00a0?\u00a0and\u00a0help()\u00a0functions in your R session. If you are using RStudio or Positron, the translated help page will appear in the usual help pane.If you are a package developer,\u00a0
lang\u00a0helps you translate your documentation, and to include it as part of your package.\u00a0lang\u00a0will use the same\u00a0?\u00a0override to display your translated help documents.
Let's look at an example. I recently invited my colleague and co-author VP Nagraj to write about the rplanes package we\u00a0published\u00a0and\u00a0released on CRAN\u00a0for plausibility analysis in epidemiological forecasting.
One of the first functions you might use from this package is\u00a0read_forecast(), which reads a probabilistic quantile forecast CSV file for downstream plausibility analysis. Let's look at the help for this function.
![\"Use](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F838aae13-ccc7-4441-bc0f-778cc6613243_1503x1156.png\")
![\"biorecap:](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6773a849-41ed-4bef-89bd-3c00f5188c87_984x999.png\")
![\"PLANES:](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58b2a85f-6645-49e8-ad87-2446bf82eaf3_1280x962.jpeg\")
library(rplanes)\n?read_forecast![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb058184-23c7-418e-9afc-dc93dc23127b_810x1154.png\")
Now let's get help in Spanish.1\u00a0load the lang package and tell it that we're using\u00a0llama3.2.2\u00a0We'll set the system language to Spanish, then ask for help again.
Sys.setenv(LANGUAGE=\"spanish\")\n?read_forecast![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F857d01c8-3b79-43a3-8bcf-81c8f8a77491_809x1111.png\")
My fluency in Spanish is limited to general conversation and travel needs so I can't easily verify the accuracy of the translation of this technical language, but when I ran some of this back through Google Translate it seemed to be mostly faithful. Notice how things that shouldn't be translated aren't \u2014 function names, arguments, columns in the returned output, code in the examples.
What about non-Western languages?
Let's try Hindi!
Sys.setenv(LANGUAGE=\"hindi\")\n?read_forecast![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c9e73f5-316e-45c2-9991-7dc3383ba997_809x1153.png\")
I can't verify the accuracy of this translation beyond running some of the text back through Google Translate, but in doing so at first glance the translation isn't bad.
What about Arabic?
Sys.setenv(LANGUAGE=\"arabic\")\n?read_forecast![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f0f290e-d033-4140-8c86-a052e5092c64_807x926.png\")
If you're a native speaker of any of these, I'd love to know what you think. Chat with\u00a0me on Bluesky (@stephenturner.us).
The\u00a0lang documentation\u00a0has a great section on using lang as a package developer. You can translate all of your Roxygen documentation into the desired language, then edit those translations by hand as needed. Then a special helper function re-roxygenizes your docs placing them in a special\u00a0inst/man-lang\u00a0folder. The\u00a0lang docs\u00a0explain how this all works, but once you do this, when a user has the lang package loaded, they'll get your pre-computed and optionally edited translations instead of having to wait around for the LLM to translate the help.
Here's a demo using a very small package I wrote for something completely different. Don't worry about all the Docker stuff described here. There's one single function,\u00a0missyelliot(), that simply reverse complements a DNA sequence (\"take that flip it and reverse it\"). That is, it'll convert GATTACA to it's reverse complement TGTAATC.
Restart your R environment, and install the package using devtools/remotes. Load both rpdd and lang.
devtools::install_github(\"stephenturner/rpdd\")\nlibrary(rpdd)\nlibrary(lang)Now get some help for\u00a0missyelliot(). If your language environment variable is English, you'll get the English help.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3963a81b-4878-4309-bf17-431650e46de6_486x590.png\")
Now, change your system language to Spanish, and try it again. Notice how the translated help is instantaneous \u2014 you're relying on the pre-translated and possibly hand-edited translations that come with the package rather than asking an LLM to translate the help for you on the fly.
Sys.setenv(LANGUAGE = \"spanish\")\n?missyelliot![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F730ae519-731b-45dd-9096-7260fd99f926_485x544.png\")
If your language is set to something without a pre-populated translation, you'll have to register a model through Ollama and translate in real time.
Sys.setenv(LANGUAGE = \"russian\")\n?missyelliot![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99937bda-121d-4858-b827-5b9cecbdb037_423x547.png\")
Reposted from the original at\u00a0https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input.
---\u00a0
If you use ChatGPT, Claude, or even some local model through\u00a0Ollama\u00a0or\u00a0HuggingFace Assistants, you'll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo. Here I'll demonstrate how to turn an entire GitHub repo into a single text file for LLM-friendly input, with R and Python packages as examples.
This\u00a0GitHub Repo to Text Converter\u00a0can help here.
Web app:\u00a0repo2txt.simplebasedomain.com
Source:\u00a0github.com/abinthomasonline/repo2txt
This little app takes a URL to a GitHub repo, lets you select which files in the repo directory structure to include, and creates a single plain text file you can feed into an LLM to ask for explanations, usage, or anything else you'd like to ask about the codebase. You can download or copy this text to the clipboard to later paste into an LLM of your choosing.
The tool runs entirely in the browser. You can also provide a personal access token if you want to access private repositories (again, securely, since everything runs in the browser).
I recently\u00a0demonstrated writing a CLI app using Click from a Cookiecutter template\u00a0in Python. It's a simple app that tells you how much caffeine remains in your system based on how much you consume and when you'll go to bed. It's on GitHub at\u00a0stephenturner/caffeinated.
Here I'm pasting the repo URL,\u00a0https://github.com/stephenturner/caffeinated\u00a0into the app. The tool is smart enough to recognize that this is a Python package, and I probably want the\u00a0__init__.py,\u00a0__main__.py, other package\u00a0.py\u00a0files, and the unit tests in\u00a0tests/\u00a0that I'll use with\u00a0pytest.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F584ba18d-fcdf-415d-aed9-2c0f5294257b_896x846.gif\")
After hitting \"Generate Text File\" you'll see that it generates a single plain text file, first listing out the directory structure of the selected files, then listing out the contents of each file separately. Here's a preview. My\u00a0__init__.py\u00a0is empty (just to signal that this is a package), and the\u00a0cli.py\u00a0file is truncated here.
![\"Build](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03154a6a-d796-4511-922f-c1391c266d53_3416x2444.png\")
Directory Structure:\n\n\u2514\u2500\u2500 ./\n \u251c\u2500\u2500 caffeinated\n \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u251c\u2500\u2500 __main__.py\n \u2502 \u2514\u2500\u2500 cli.py\n \u2514\u2500\u2500 tests\n \u2514\u2500\u2500 test_caffeinated.py\n\n\n---\nFile: /caffeinated/__init__.py\n---\n\n\n---\nFile: /caffeinated/__main__.py\n---\n\nfrom .cli import caffeinated\n\nif __name__ == \"__main__\":\n caffeinated()\n\n\n---\nFile: /caffeinated/cli.py\n---\n\nimport click\nimport math\nfrom datetime import datetime\nfrom importlib.metadata import version, PackageNotFoundError\n\n### TRUNCATED ###Once you have this, you can easily paste this into an LLM of your choice to chat with the codebase. Here, I pasted this into GPT-4o with the leading prompt:\u00a0\"Explain this code to me, how to use it what it does.\"\u00a0GPT-4o returned all of the text and code below as a single response.
First, the directory structure.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba7e380f-5c28-423f-9536-1edbe767621a_824x528.png\")
Next, an explanation of\u00a0__main__.py,\u00a0cli.py, and the tests.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047cf148-24da-43f7-8ec6-70b53534bbeb_720x1072.png\")
Next, a demonstration of how to use it. Note above that I did\u00a0not\u00a0select the README to add to my text file. GPT-4o divined this usage information from the code itself, rather than regurgitating the README.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d7a503e-ed47-4de2-b5b4-866067ff741d_781x1104.png\"){kind=spacer}
And finally, a simple explanation of the (minimal) test suite.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F454398aa-b7c8-4efe-9219-13e251ae6fa6_752x408.png\")
This also works on an R package. In my last job I wrote a package to assist with containerizing R packages with a usethis-like interface. It's called pracpac (practical R packaging [with Docker]), and you can read more in the\u00a0paper\u00a0or on the\u00a0GitHub repo. Here I'll take the code from\u00a0pracpac repo\u00a0and pull out the relevant R source files.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e6fd684-47a6-4a45-9a6f-840f28b04941_934x1060.png\")
This time I asked Claude 3.5 Sonnet to tell me more about this package.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2181bd0a-08db-48a1-9597-2a6f2009a14c_865x1285.png\")
If you look at the\u00a0source code\u00a0you'll see that this is a simple implementation using just HTML and JavaScript. Which means you can download the repo and open the index.html, or if you'd prefer a vanity URL, you can fork the repo and set up GitHub pages to serve from your main branch (e.g.\u00a0stephenturner.github.io/repo2txt).
","doi":"https://doi.org/10.59350/3t3wg-g6750","guid":"tag:blogger.com,1999:blog-6232819486261696035.post-352146836295256074","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1733702400,"rid":"qd318-eke39","summary":" Reposted from the original at https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input. --- If you use ChatGPT, Claude, or even some local model through Ollama or HuggingFace Assistants, you'll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo.","tags":["R"],"title":"Turn a GitHub repo into a single text file for LLM-friendly input (repost)","updated_at":1788597424,"url":"https://gettinggeneticsdone.blogspot.com/2024/12/github-repo-to-text-for-llm-input.html","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"Reposted from the original at\u00a0https://blog.stephenturner.us/p/python-cli-click-cookiecutter.\u00a0
---
In the spirit of\u00a0Learning in Public, I wanted an excuse to explore (1)\u00a0click\u00a0for creating command line interfaces, (2)\u00a0Cookiecutter\u00a0project templates, and (3) modern tools in the Python packaging ecosystem. If you're primarily an R developer like me, I recently wrote about resources for getting better at Python for R users.
Click\u00a0is a really nice package for creating command line interfaces, and I like it better than argparse or other similar utilities. Simon Willison's\u00a0click app cookiecutter template\u00a0was really helpful in getting the boilerplate set up for a python package, and while I looked at build backends like Flit, Poetry, uv, etc., I ended up just using setuptools with a pyproject.toml.
For this demo I built a silly little Python tool called\u00a0caffeinated\u00a0(inspired by\u00a0coffee-o-clock) that tells you how much caffeine you'll still have in your system at bedtime based on how much you consume and when. You can install it from\u00a0PyPI\u00a0and running\u00a0caffeinated\u00a0with the\u00a0--help\u00a0option (or without any arguments) prints the help. The code is on GitHub (github.com/stephenturner/caffeinated) if you want to follow along.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03154a6a-d796-4511-922f-c1391c266d53_3416x2444.png\")
![\"Python](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2049d794-5c0e-4bde-a0a9-f51d86413057_1248x920.png\")
pip install caffeinated, then run\u00a0caffeinated --help\u00a0for usage info. Run\u00a0caffeinated -c 200 -b 9pm\u00a0to see how much caffeine will remain in your system if you consume 200mg caffeine right now and go to bed at 9pm.I'm not totally sure how accurate the formula is here, but I'm using this to calculate how much caffeine remains in circulation (and I'm going with 90mg for \"a cup of coffee\").1
Where:
N(t)\u00a0= Quantity of caffeine remaining
N0\u00a0= Original amount of caffeine
t\u00a0= Time
t6\u00a0= Coffee's half-life (6 hours)
Click (Command Line Interface Creation Kit) is a Python package for creating command line interfaces in a composable way with as little code as necessary.
Why Click? Why not argparse/docopt/etc?\u00a0Good questions. The Click documentation has a section on\u00a0Why Click?,\u00a0Why not Argparse?\u00a0and\u00a0Why not Docopt etc.?\u00a0I like click because it enables you to easily create command line utilities with subcommands (e.g.,\u00a0mycommand subcommand ...\u00a0e.g. like\u00a0bedtools intersect ...), and it supports file handling, makes it easier to handle options versus arguments, and easily supports ANSI coloring of the output.
First let's set up a folder structure we'll use to create a Python package. Make a new folder named whatever you're calling the package (in this case,\u00a0caffeinated), and in that directory create a\u00a0pyproject.toml\u00a0file. Create a new subfolder with the same name as the parent directory (caffeinated), and in that folder you'll have three files. Directory structure should look like this:
caffeinated/
caffeinated/
__init__.py
__main__.py
cli.py
pyproject.toml
The pyproject.toml will have just the basics you need for a Python package:
[project]\nname = \"caffeinated\"\nversion = \"0.1.1\"\ndependencies = [\"click\"]\n\n[project.scripts]\ncaffeinated = \"caffeinated.cli:caffeinated\"The\u00a0__init__.py\u00a0will be empty, and the\u00a0__main__.py\u00a0will just have one line that imports the function from the cli.py:
from .cli import caffeinated\n\nif __name__ == \"__main__\":\n caffeinated()The\u00a0cli.py\u00a0actually has the code for your command line tool. This is a really simple program that just echos out the amount of caffeine you consumed and what time your bedtime is:
import click\n\n@click.command()\n@click.option(\"-c\", \"--caffeine\", default=100)\n@click.option(\"-b\", \"--bedtime\", default=2100)\ndef caffeinated(caffeine, bedtime):\n click.echo(f\"Caffeine consumed: {caffeine} mg\")\n click.echo(f\"Your bedtime is: {bedtime}\")Now, pip install the package you just wrote:
pip install .And now the caffeinated command line utility is ready to use. First, get some help. Notice how by specifying a default value
$ caffeinated --help\nUsage: caffeinated [OPTIONS]\n\nOptions:\n -c, --caffeine INTEGER\n -b, --bedtime INTEGER\n --help Show this message and exit.Now run it:
$ caffeinated -c 200 -b 2100\nCaffeine consumed: 200 mg\nYour bedtime is: 2100You can see the real cookiecutter code here:\u00a0github.com/stephenturner/caffeinated. Here are links to the actual working code. Everything important is in the\u00a0cli.py\u00a0file. It adds a few more arguments, picks up the version from the pyproject.toml, and defines functions to do all the calculation and conveniences such as translating \"9pm\" into 2100 (hours).
Once you update all the source or just\u00a0pip install caffeinated\u00a0again from PyPI, the tool will tell you approximately how much caffeine you'll have remaining in your system after consuming a certain amount of caffeine at your chosen bedtime. Run\u00a0caffeinated --help\u00a0to get help on the options.
Organizing your project as a well-structured Python package can streamline development and distribution. The\u00a0Cookiecutter\u00a0package provides a straightforward way to generate project templates, ensuring consistency and best practices across your projects. Install it with pip, then I'll use\u00a0Simon Willison's click-app cookiecutter template.
pip install cookiecutter\ncookiecutter gh:simonw/click-appYou can see what this looks after running that at\u00a0this demo. Running this and answering a few of the prompts will create:
The directory structure described above
The\u00a0__init__.py,\u00a0__main__.py, and\u00a0cli.py\u00a0files with some boilerplate to get started.
A pyproject.toml file based on your answers to the prompts.
A tests directory with boilerplate for writing tests with pytest.
A README with badges pointing to a future PyPI release, changelog from your GitHub releases, license, and test status.
GitHub actions for publishing your tool as a package to PyPI (requires additional configuration\u00a0as described here).
The final caffeinated code made from Simon's click-app cookiecutter template is here\u00a0github.com/stephenturner/caffeinated.
I've written scores of R packages for fun and profit. There's really only one build backend toolchain for R packages that everyone uses:\u00a0devtools\u00a0with\u00a0Roxygen\u00a0documentation with liberal assistance from\u00a0usethis.
The Python documentation has a good guide on\u00a0Packaging Python Projects. The build backend ecosystem in Python is more diverse. I really wanted to take a closer look at\u00a0Flit,\u00a0Poetry,\u00a0Hatch, and others, but because the cookiecutter template that I used created a pyproject.toml using\u00a0setuptools\u00a0by default, so I just ran with that.
setuptools\u00a0is probably the oldest and most widely used packaging tools in Python with good documentation and community support. And with\u00a0PEP 517, the standard became using a simpler pyproject.toml rather than the old setup.py. You can see my\u00a0pyproject.toml for caffeinated here. It's pretty simple, and one key feature is the\u00a0readme=\"README.md\"\u00a0entry, which results in the documentation on the PyPI landing page (pypi.org/project/caffeinated/) populated with the README.md in the project root, avoiding the need for duplication.
Flit\u00a0(flit.pypa.io) looks like a very minimal, very simple build backend for packaging plain Python code. I also took a look at\u00a0Poetry\u00a0(python-poetry.org), because if I were building something more complex I think I'd want something to help me manage dependencies instead of having to add them to the pyproject.toml by hand. Poetry helps with this. Finally, there's a lot of interest in\u00a0uv\u00a0(docs.astral.sh/uv) right now. It's a Python package and project manager written in Rust, and the benchmarks are impressive. See the\u00a0\"uv: Unified Python packaging\"\u00a0blog post for more. uv doesn't yet have a build backend, but that's in the works at\u00a0astral-sh/uv#3957.
Building python packages with setuptools vs Flit vs Poetry vs uv might be the subject of a future post, but for now, I'm just using setuptools+build.
The pyproject.toml created by the cookiecutter template I used had just about everything I needed to build the package. From here it was simple. This will build the\u00a0.whl\u00a0binary file and\u00a0.tar.gz\u00a0source packages in a\u00a0dist/\u00a0folder.
python -m buildAfter this it's fairly straightforward to upload this to PyPI. But, before uploading to the production\u00a0pypi.org, you should probably upload to the testing repository (test.pypi.org) first to avoid polluting PyPI with broken or testing packages.
twine upload -r testpypi dist/*Uploading to the real PyPI follows the same convention. I recommend using a\u00a0token in your\u00a0.pypirc\u00a0file\u00a0instead of a username/password prompt.
twine upload dist/*A few seconds later your package will be on PyPI, and you can install it with pip install like you would any other package. The caffeinated package is on PyPI at\u00a0https://pypi.org/project/caffeinated/.
If you've ever tried getting a package into CRAN you know how onerous the process can be and how strict the CRAN maintainers can be2\u00a0and you may be shocked to see how easy it is to get a package onto PyPI. There is no curation or review process with PyPI. Upload your source and wheel files with twine and your package is live.
Once the app is on PyPI it's easy to create a Docker container. There's a Dockerfile in the repo that looks like this:
FROM python:3.11-alpine\nRUN pip install caffeinated\nWORKDIR /files\nENTRYPOINT [\"caffeinated\"]\nCMD [\"--help\"]You can build it like this (replace stephenturner with your Docker username):
docker build -t stephenturner/caffeinatedAnd push it up to Docker Hub:
docker push stephenturner/caffeinated This container is on Docker hub at\u00a0stephenturner/caffeinated. Now you can run it:
$ docker run stephenturner/caffeinated -c 200 -b 9pm\n\nYou would have 35.4mg of caffeine in your system if you went to bed at 9:00pm (in 15.0 hours).\nThat's like having 39% of a cup of coffee before bed.Alternatively, you can easily create an image with your new tool and additional tools you might want in the same container using\u00a0Seqera containers\u00a0(see the video linked further below for more details). I created\u00a0this image\u00a0that includes both caffeinated and\u00a0cowsay, and I'm piping the result of caffeinated into cowsay.
$ docker run --rm community.wave.seqera.io/library/pip_caffeinated_python-cowsay:5a33eb2abfe4e6a5 sh -c 'caffeinated --caffeine 200 --bedtime 9pm --start-time 8am | cowsay'\n __________________________________________ \n/ You would have 44.5mg of caffeine in \\\n| your system if you went to bed at 9:00pm |\n| (in 13.0 hours). That's like having 49% |\n\\ of a cup of coffee before bed. /\n ------------------------------------------ \n \\ ^__^\n \\ (oo)\\_______\n (__)\\ )\\/\\\n ||----w |\n || ||This is part of my TIL / Learning in Public series, which I wrote about recently:
For a quick intro on Click, see the official Click intro video. It's over 10 years old at this point, it's mostly still applicable.
This video from NeuralNine demonstrates how to create Click groups to create CLI utilities with subcommands:
This video from ArjanCodes explains how to create a Python Package and publish on PyPI. It uses the old setup.py instead of the more modern pyproject.toml convention, but otherwise it's still good to understand the steps in the process.
Finally, a little more about easily creating containers with multiple tools for multiple architectures using\u00a0Seqera Containers:
\u00a0This is reposted from the original at\u00a0https://blog.stephenturner.us/p/python-for-r-users.
---
A Google search for \"R vs Python\" returns thousands of hits across sites like Reddit, IBM, Datacamp, Coursera, Kaggle, and many others. A quick Google Trends analysis shows that this search query has grown steadily over the last decade.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb18733-39a2-463b-a2e6-dab4368c216a_650x400.png\")
Any real data scientist would agree that this argument is silly, that the right answer is to use the best tool for the job. What's \"best\" isn't always easy to answer. Sometimes the \"best tool\" might be the one that you're most familiar with and can write most easily. After all, compute time is cheap relative to software engineer time. In other cases there truly is a better choice depending on the problem (e.g., the AI ecosystem is generally more well developed in Python, while the Bioconductor ecosystem provides R with a rich framework for many kinds of bioinformatics tasks).\u00a0I'm partial to R/tidyverse for tabular data analysis and manipulation, but if polars or pandas is your thing, that's cool too.
Beyond \"choose the right tool,\" my biggest advice to trainees and students when asked about \"R vs Python\" was to learn both languages,\u00a0but learn one of them so well as to be native-level\u00a0fluent\u00a0in the language. That is, become so comfortable in a language that you rarely have to look up documentation, where you have a natural intuition about how to best implement a variety of your everyday tasks, where you know how to write code, docs, and unit tests without really thinking about it.
R is that language for me. I can read, write, debug, and package code in Python, but it doesn't come to me as easily. I've been collecting and bookmarking resources over the years that I've turned to for writing and reading Python, coming from native R fluency. LLMs have a new place in this as well, and I touch on that in the end.
If you're just getting started with Python or programming in general, there are thousands of courses and resources to help you get started. My favorite one to recommend is David Malan's CS50 course at Harvard. The course materials are\u00a0here, and the entire 16-hour course is available on YouTube. I still revisit the OOP section from time to time. It likely covers concepts you already know well, as this is meant to be an introduction to programming, using Python.
https://www.youtube.com/watch?v=nLRL_NcnK-4
learnxinyminutes.com has a collection of very short primers on different programming languages. These assume you've programmed in other languages before and understand flow control, OOP, functions, classes, etc. The\u00a0Learn X in Y minutes where X=Python\u00a0tour is here:\u00a0learnxinyminutes.com/docs/python/. It's a good place to start if you don't need to learn what a for loop is or how flow control works. This is simply a long Python scripts with comments throughout that demonstrate main features of the language for programmers coming from another language.
Reticulate\u00a0is an R package that enables interoperability between R and Python. It allows you to call Python from R, translate between R and python objects, and create bindings to different Python virtual environments or Conda environments.
The\u00a0Primer on Python for R Users vignette\u00a0in the reticulate package is a great place to start if you're a seasoned R user and want to know how to write Python code to do things in R that you're familiar with. The primer covers data structures, iteration, indexing, function definition, classes, dunder methods, iterators, generators, imports and modules, decorators, and more.
I've never personally met Emily but I've been following her since I saw her rstudio::global 2020 talk on building an R package ecosystem internal to your organization (video,\u00a0slides,\u00a0post). It's a great talk whether you're programming in R, Python, Rust, whatever, and building off of open-source tools packages for internal organization use only.
Emily recently published a three part series on Python ergonomics and patterns for seasoned R developers. Links to her posts with her description below.
Python Rgonomics: Switching languages is about switching mindsets - not just syntax. New developments in python data science toolings, like polars and seaborn's object interface, can capture the 'feel' that converts from R/tidyverse love while opening the door to truly pythonic workflows.
polars' Rgonomic Patterns: In this follow-up post to Python Rgonomics, we deep dive into some of the advanced data wrangling functionality in python's\u00a0polars\u00a0package to see how it's powertools like column selectors and nested data structures mirror the best of\u00a0dplyr\u00a0and\u00a0tidyr's expressive and concise syntax
Base Python Rgonomic Patterns: Getting comfortable in a new language is more than the packages you use. Syntactic sugar in base python increases the efficiency, and aesthetics of python code in ways that R users may enjoy in packages like\u00a0glue\u00a0and\u00a0purrr. This post collects a miscellaneous grab bag of tools for wrangling, formatting (f-strings), repeating (list comprehensions), faking data, and saving objects (pickle).
Additionally, I wasn't able to make it to posit::conf(2024) this year, but from a few colleagues who did, I heard her talk on the topic was great. The talks from the conference aren't yet online, but Posit usually posts them to YouTube a few months after the conference. Description of her talk from the\u00a0conference agenda:
Python Rgonomics
Data science languages are increasingly interoperable with advances like Arrow, Quarto, and Posit Connect. But data scientists are not. Learning the basic syntax of a new language is easy, but relearning the ergonomics that help us be hyperproductive is hard. In this talk, I will explore the influential ergonomics of R's tidyverse. Next, I will recommend a curated stack that mirrors these ergonomics while also being genuinely truly pythonic. In particular, we will explore packages (polars, seaborn objects, greattables), frameworks (Shiny, Quarto), dev tools (pyenv, ruff, and pdm), and IDEs (VS Code extensions). The audience should leave feeling inspired to try python while benefiting from their current knowledge and expertise.
Hadley, Mine, and Garrett's book R for Data Science is a must-read for anyone looking to upskill their R chops for data analysis and visualization, and the second edition is available to read free online at\u00a0r4ds.hadley.nz.
Arthur Turrell\u00a0and others have written\u00a0Python for Data Science\u00a0(aeturrell.github.io/python4DS), heavily inspired by R4DS, with most of the chapters mirroring R4DS. The image below shows the TOCs from both books. The book mostly uses pandas and lets-plot (\"ggplot2 for python\").
![\"\"](\"https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fa0d953-512f-4fe9-a1dc-76e9de361eca_491x1306.png){kind=spacer}
The Shiny for Python docs (shiny.posit.co/py/docs) are pretty good for getting into Shiny for Python, especially if you have some experience with Shiny for R.
If you're looking for more of a hand-holding approach, I'd recommend\u00a0Gordon Shotwell's free\u00a0Reactive Web Dashboards with Shiny Course\u00a0available on Talk Python Training.
You don't get better at Python by reading about Python or watching videos about Python. There's no lack of good resources for Python exercises. A few resources that I've found particularly useful are listed below.
Corey Schafer's YouTube channel (youtube.com/@coreyms): This channel has playlists of short videos and tutorials on Pandas, Matplotlib, OOP, and many other short and long video walkthroughs I've found useful.
Python on Exercism (exercism.org/tracks/python): Lots of free exercises.
Practice Python (practicepython.org):\u00a0More practice!
Automate the Boring Stuff with Python (automatetheboringstuff.com): A free online book with simple projects for automating boring tasks (renaming files, etc).
Advent of Code (adventofcode.com): An Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. It starts every year around the holidays, but you can go back and view old puzzles from previous years any time. These are fun for little friendly office competition as well.
Of all the advice in this post, this one is likely to get stale fastest, so I'll keep it short. Use LLMs where you can! ChatGPT and other LLMs can essentially act as your pair programmer or personal StackOverflow helping you translate your R expertise into Python with contextualized explanations. You can ask ChatGPT for Python equivalents of familiar R functions, syntax comparisons, or specific code examples tailored to data manipulation, statistical analysis, or visualization. You can paste in Python code, ask what it's doing, and see what equivalent R code would have been. Paste in error messages and your code and it'll tell you what went wrong and how to change things. I imagine all the leading LLMs are better with Python than nearly any other programming language, so make use of these tools where you can!
![\"\"](\"https://blogger.googleusercontent.com/img/a/AVvXsEig_J_7Cjyc3m0HX9Uv2KX_uGl2yadoJE2I3wuxQ8Kzw_ZDT-5wuEiZyv2VjcJfc_kDoKamErZEZs5EW17Ek8aIY8q-Yy1VprWRh4KjSEMx0RWgzemcdu5mldcKkr-qdaIgm6X98mxcqjNVSON4gtLtmfiZvsn4bV_eIwni80Ygu3CeLg6sHeWf3kCWf38=w378-h279\")
\u00a0This is reposted from the original at\u00a0https://blog.stephenturner.us/p/use-nanoparquet-instead-of-readr-csv.
Yesterday I wrote about base R vs. dplyr vs. duckdb for a simple summary analysis. In that post I simulated 100 million rows of a dataset and wrote to disk as CSV. I then benchmarked how long it took to read in and compute a simple grouped mean.
One thing I\u00a0didn't\u00a0do here was separate the time it took to read data into memory (for base R and dplyr) versus computing the actual summary. Remember, with DuckDB you don't actually have to read the data into memory for SQL operations you can do with DuckDB or duckplyr.
After I wrote this I downloaded the materials from Hadley Wickham's\u00a0R in Production workshop from posit::conf(2024)\u00a0where he encourages using parquet instead of CSV for saving and sharing data.
Why parquet? From the slides:
Like csv, every language can read and write.
Binary column-oriented format (much faster to read).
Rich type system (including missing values).
Efficient encodings + optional compression (smaller files).
Data stored in \"chunks\" (can work in parallel & don't need to read entire file).
Supports complex data types (e.g. list-columns).
Many modern databases can use parquet files directly.
In these slides I also learned about the\u00a0nanoparquet\u00a0package\u00a0\u2014 a\u00a0zero dependency\u00a0package for reading and writing parquet files in R. Besides all the benefits noted above, parquet is much faster to read and write. And, as opposed to saving as .rds, parquet can easily be passed back and forth between R, Python, and other frameworks.
Let's take a look at how reading and writing parquet files compares with CSV, either with base R or readr.
First, load the libraries I'll use here.
![\"Learning](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fe7c690-01f5-4313-8186-e8f969837b48_1656x1176.png\")
![\"DuckDB](\"https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5277de18-0343-4ad1-b719-641106652e39_2100x1500.png\")
library(tidyverse)\nlibrary(nanoparquet)Next, simulate 100 million rows, 5 columns of numeric, categorical, and date data.
n <- 1e8\nset.seed(42)\ndat <-\n tibble(\n id = sample(n/10, n, replace = TRUE),\n category = sample(letters, n, replace = TRUE),\n value1 = rnorm(n),\n value2 = runif(n, min = 0, max = 1000),\n date = sample(seq.Date(from = as.Date(\"2010-01-01\"),\n to = as.Date(\"2020-12-31\"),\n by = \"day\"),\n size=n,\n replace = TRUE)) |>\n arrange(id, category, date)\nThe data looks like this:
# A tibble: 100,000,000 \u00d7 5\n id category value1 value2 date\n \n 1 1 b -1.26 145. 2020-02-15\n 2 1 g -0.316 157. 2020-06-03\n 3 1 i -1.42 919. 2019-11-24\n 4 1 k 0.592 349. 2011-06-15\n 5 1 l -0.292 650. 2013-03-19\n 6 1 m 0.316 802. 2015-05-25\n 7 1 q 0.112 470. 2020-07-13\n 8 1 t 0.637 236. 2011-02-26\n 9 1 w 1.72 49.5 2016-02-25\n10 2 a -0.572 506. 2018-01-22\n# \u2139 99,999,990 more rows Next, I'll create a list holding the time it takes to read and write using base R, readr, and nanoparquet. Notice that readr will attempt to guess the column types when reading in data, by default using only the first 1,000 rows. This could lead to problems if your first 1,000 rows contained numeric values, but later on in the file you have strings or other data types.
time <- list()\ntime$write.csv <- system.time(write.csv(dat, \"dat.csv\"))\ntime$write_csv <- system.time(write_csv(dat, \"dat.csv\"))\ntime$write_pqt <- system.time(write_parquet(dat, \"dat.parquet\"))\ntime$read.csv <- system.time(read.csv(\"dat.csv\"))\ntime$read_csv <- system.time(read_csv(\"dat.csv\"))\ntime$read_pqt <- system.time(read_parquet(\"dat.parquet\"))Finally, extract the timing into a data frame, then plot.
time |>\n map_dbl(\"elapsed\") |>\n enframe(\"fn\", \"seconds\") |>\n mutate(task=ifelse(grepl(\"write\", fn), \"write\", \"read\")) |>\n mutate(pkg=case_when(\n grepl(\"\\\\.csv$\", fn) ~ \"base_R\",\n grepl(\"_csv$\", fn) ~ \"readr\",\n grepl(\"_pqt$\", fn) ~ \"nanoparquet\"\n )) |>\n mutate(pkg=fct_relevel(pkg, \"base_R\", \"readr\", \"nanoparquet\")) |>\n ggplot(aes(task, seconds, fill=pkg)) +\n geom_col(position=\"dodge\") +\n theme_classic() +\n labs(x=\"Task\", y=\"Time (seconds)\", fill=NULL) + \n theme(legend.position=\"bottom\")The results are compelling:
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5684d64-19f5-4abe-b6c3-e5b2e9ccd29d_2100x1500.png\" "\\"\\"")
In tabular form:
task base_R readr nanoparquet\nread 366 21 6\nwrite 231 68 8Writing with nanoparquet is 29x faster than base R, and 8.5x faster than readr. Reading with nanoparquet is 61x faster than base R, 3.5x faster than readr. With the added bonus of column types coming in as the types they were originally created as (id as integer, the date column as an actual date). Additionally, the CSV is 5.5 Gb on disk (which could be compressed), and the parquet file is only 1.9 Gb.
As noted above, another nice feature of using parquet is the ability to store column types as they were originally created, rather than relying on\u00a0readr::read_csv()\u00a0to correctly guess the column type (or supply column types explicitly).
The nanoparquet package has a few other functions that allow us to see various kinds of metadata about the parquet files on disk without actually reading them in. From the\u00a0documentation:
parquet_info()\u00a0shows a basic summary of the file.
parquet_column_types()\u00a0shows the leaf columns, these are are the ones that\u00a0read_parquet()\u00a0reads into R.
parquet_schema()\u00a0shows all columns, including non-leaf columns.
parquet_metadata()\u00a0shows the most complete metadata information: file meta data, the schema, the row groups and column chunks of the file.
> parquet_info(\"dat.parquet\")\n# A data frame: 1 \u00d7 7\n file_name num_cols num_rows ...\n ...\n1 dat.parquet 5 100000000 ...\n\n> parquet_column_types(\"dat.parquet\")\n# A data frame: 5 \u00d7 5\n file_name name type r_type logical_type\n* >\n1 dat.parquet id INT32 integer \n2 dat.parquet category BYTE_ARRAY character \n3 dat.parquet value1 DOUBLE double \n4 dat.parquet value2 DOUBLE double \n5 dat.parquet date INT32 Date Similar to what was demonstrated in the previous post, we can run a query on this data to summarize values grouped by ID for all ~10 million groups in this data using duckplyr as an interface to DuckDB,\u00a0without ever having to actually read the parquet files into memory. Here's how to do that (the\u00a0nrow()\u00a0at the end actually materializes the resulting data frame into a tibble in your R session).
Similar to the\u00a0previous post, this summary analysis took about\u00a03.2 seconds\u00a0on 100 million rows, without actually having to read the data into memory.
library(duckplyr)\nsystem.time({\n res <- \n duckplyr_df_from_parquet(\"dat.parquet\") |>\n summarize(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2),\n .by=id) |>\n arrange(id)\n nrow(res)\n})Results preview:
# A tibble: 9,999,521 \u00d7 3\n id mean_value1 mean_value2\n \n 1 1 0.00983 420.\n 2 2 -0.379 661.\n 3 3 -0.0515 491.\n 4 4 0.0574 542.\n 5 5 -0.173 398.\n 6 6 0.0854 405.\n 7 7 0.200 510.\n 8 8 0.0117 621.\n 9 9 -0.0726 470.\n10 10 -0.456 503.\n# \u2139 9,999,511 more rows If you'd rather write raw SQL instead of using duckplyr, you can do that too. This also takes 3.2 seconds for 100 million rows.
library(DBI)\nlibrary(duckdb)\nsystem.time({\n conn <- dbConnect(duckdb())\n res <-\n dbGetQuery(conn = conn,\n statement=\"\n SELECT id,\n AVG(value1) as mean_value1,\n AVG(value2) as mean_value2\n FROM read_parquet('dat.parquet')\n GROUP BY id\n ORDER BY id\")\n dbDisconnect(conn, shutdown=TRUE)\n})
Reposted from\u00a0https://blog.stephenturner.us/p/duckdb-vs-dplyr-vs-base-r.
TL;DR: For a very simple analysis (means by group on 100M rows), duckdb was 125x faster than base R, and 28x faster than readr+dplyr, without having to read data from disk into memory. The duckplyr package wraps DuckDB's analytical query processing techniques in a dplyr-compatible API. Learn more at\u00a0duckdb.org/docs/api/r\u00a0and\u00a0duckplyr.tidyverse.org.
I wanted to see for myself what the fuss was about with DuckDB.\u00a0DuckDB\u00a0is an open-source, high-performance, in-process SQL query engine designed specifically for analytical tasks. Often referred to as the \"sqlite for analytics\" DuckDB is optimized for OLAP (Online Analytical Processing) workloads \u2014 data is organized by columns meaning that the database is optimized for fast reading and large, complex queries that involve filtering, aggregations, and joins. It's \"embedded\" in the same sense that sqlite is \u2014 it has no external dependencies, no DBMS server to install, setup, and maintain. It doesn't as a separate process \u2014 it runs embedded within a host process, conferring the advantage of high-speed data transfer to and from the database, or in some cases direct querying of the data without actually importing or copying any data at all. Read more at\u00a0duckdb.org/why_duckdb.
DuckDB has an R API available (see\u00a0docs). You can install the duckdb package from\u00a0CRAN\u00a0or\u00a0GitHub. Why might you want to use DuckDB in R?
Large Datasets: When you have data too large for memory, DuckDB can efficiently query data from disk without needing the entire dataset in memory.
SQL-Like Queries: If you're more comfortable with SQL, DuckDB allows you to run SQL queries directly on your R data frames. You can perform complex operations such as grouping, filtering, joining, and aggregating, using familiar SQL syntax. Again, without ever reading data into memory in R.
Efficient Analytical Queries: For workloads involving large aggregations or queries on millions of rows DuckDB can handle large analytical queries while minimizing memory overhead and keeping performance high.
Stay in R!\u00a0Using DuckDB in R lets you use DuckDB for the heavy lifting analytical queries, giving you results back in a table that you can take forward for visualization or further analysis.
I wanted to get a sense for myself how fast DuckDB might be compared to base R and dplyr. This will be a simple experiment: I'll simulate some data and write out to disk, then I'll use base R, dplyr, and duckdb to read in the data, group by a variable, and compute the means of two different columns. Simple. There are other, better benchmarks elsewhere (e.g., see posts\u00a0here\u00a0and\u00a0here\u00a0from Appsilon).
All the code below requires a few libraries loaded.
library(tidyverse)\nlibrary(DBI)\nlibrary(duckdb)The code below simulates 100 million rows and some random values and dates.
n <- 1e8\nset.seed(42)\ndat <-\n tibble(\n id = sample(1:1e7, n, replace = TRUE),\n category = sample(letters, n, replace = TRUE),\n value1 = rnorm(n),\n value2 = runif(n, min = 0, max = 1000),\n date = sample(seq.Date(from = as.Date(\"2010-01-01\"),\n to = as.Date(\"2020-12-31\"),\n by = \"day\"), n, replace = TRUE)) |>\n arrange(id, category, date)\ndat |> write_csv(\"dat.csv\")\nThe first 15 rows looks like this:
id category value1 value2 date \n \n 1 1 b -1.26 145. 2020-02-15\n 2 1 g -0.316 157. 2020-06-03\n 3 1 i -1.42 919. 2019-11-24\n 4 1 k 0.592 349. 2011-06-15\n 5 1 l -0.292 650. 2013-03-19\n 6 1 m 0.316 802. 2015-05-25\n 7 1 q 0.112 470. 2020-07-13\n 8 1 t 0.637 236. 2011-02-26\n 9 1 w 1.72 49.5 2016-02-25\n10 2 a -0.572 506. 2018-01-22\n11 2 c -0.608 440. 2018-10-13\n12 2 c -1.26 553. 2020-09-25\n13 2 g 0.563 773. 2018-02-27\n14 2 l 1.07 933. 2012-07-29\n15 2 l -0.163 363. 2017-10-07 The three functions below takes a file path, reads in the data1, computes the mean of value1 and value2 grouped by id, and returns a list of the summarized result and the time it took to run the analysis.
With base R:
process_base <- function(x) {\n time <- system.time({\n dat <- read.csv(x)\n res <- aggregate(cbind(value1, value2) ~ id, data = dat, FUN = mean)\n res <- res[order(res$id), ]\n })\n return(list(res=res, time=time))\n}With dplyr:
process_dplyr <- function(x) {\n time <- system.time({\n dat <- read_csv(x, show_col_types=FALSE, progress=FALSE)\n res <- dat %>%\n group_by(id) %>%\n summarise(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2)\n )\n })\n return(list(res=res, time=time))\n}With duckdb (see below for a friendlier way to write this with duckplyr):
process_duckdb <- function(x) {\n time <- system.time({\n conn <- dbConnect(duckdb())\n res <-\n dbGetQuery(conn = conn,\n statement=glue::glue(\"\n SELECT id,\n AVG(value1) as mean_value1,\n AVG(value2) as mean_value2\n FROM read_csv('{x}', parallel=true)\n GROUP BY id\n ORDER BY id\"))\n })\n dbDisconnect(conn, shutdown=TRUE)\n return(list(res=res, time=time))\n}The results were striking, shown below. Using duckdb was 125x faster than base R, and 28x faster than readr+dplyr.
duckdb: 3.86 seconds
dplyr: 110 seconds
base: 486 seconds
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5277de18-0343-4ad1-b719-641106652e39_2100x1500.png\")
The DBI +\u00a0dbConnect()\u00a0+\u00a0dbGetQuery()\u00a0with plain SQL above isn't the most ergonomic or idiomatic R way to read and analyze data in R. Good news: the\u00a0duckplyr\u00a0package for R wraps DuckDB's analytical query processing techniques in a dplyr-compatible API. Learn more about it at\u00a0duckplyr.tidyverse.org.
Here I'm using the\u00a0duckplyr_df_from_file()\u00a0function to \"read in\" the data from a file, while the summarize and arrange methods are overwriting the functions from dplyr with the duckplyr version. This operation also takes ~3 seconds, and is much easier to read with the familiar dplyr tools you're used to. Take a look at\u00a0DuckDB's blog post on duckplyr\u00a0to really understand how this works.
library(tidyverse)\nlibrary(duckplyr)\nres <-\n duckplyr_df_from_file(\"dat.csv\", table_function=\"read_csv\") |>\n summarize(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2),\n .by=id) |>\n arrange(id)\nresI recently wrote a short essay on\u00a0Learning in Public.\u00a0Something I'm trying to do more of.
The \"benchmark\" here was a simple and limited one, and I mostly used it as an excuse for me to learn a little about duckdb and how to integrate it into my workflow. In this benchmark I didn't attempt to separate time spent reading data into memory versus time spent processing, and the \"analysis\" was a simple mean by group. I didn't attempt to parallelize or optimize either the base R or dplyr approaches, and I didn't do anything with data.table.
For more extensive benchmarks I'd recommend reading through a few of Appsilon's recent posts on using duckdb in R, for huge data, and with Shiny:
There are also a few talks from last year's posit::conf(2023) on using R with duckdb. Those talks are available on YouTube.
Finally, Hannes M\u00fchleisen (CEO, DuckDB Labs) gave a keynote at this year's posit::conf(2024):\u00a0Data Wrangling [for Python or R] Like a Boss With DuckDB. The talks from this year's conference aren't out yet. The talk's abstract from the\u00a0conference agenda\u00a0is below.
Data wrangling is the thorny hedge that higher powers have placed in front of the enjoyable task of actually analyzing or visualizing data. Common struggles come from importing data from ill-mannered CSV files, the tedious task of orchestrating efficient data transformation, or the inevitable management of changes to tables. Data wrangling is rife with questionable ad-hoc solutions, which can sometimes even make things worse. The design rationale of DuckDB is to support the task of data wrangling by bringing the best of decades of data management research and best practices to the world of interactive data analysis in R or Python. For example, DuckDB has one of the world's most advanced CSV reader, native support for Parquet files and Arrow structures, an efficient parallel vectorized query processing engine, and support for efficient atomic updates to tables. All of this is wrapped up in a zero-dependency package available in a programming language near you for free. In my talk, I will discuss the above as well as the design rationale of DuckDB, which was designed and built in collaboration with the Data Science community in the first place.
Keep an eye out for whenever Posit announces the recordings are publicly available.
In the meantime, the workshop materials from posit::conf(2024) are available, including the \"Databases in R\" workshop, taught by Kirill M\u00fcller.
Workshop materials:\u00a0https://github.com/posit-conf-2024/databases
Slides part 1:\u00a0https://posit-conf-2024.github.io/databases/
Slides part 2:\u00a0https://posit-conf-2024.github.io/databases/part-2
Slides part 3:\u00a0https://posit-conf-2024.github.io/databases/part-3.html
Slides part 4:\u00a0https://posit-conf-2024.github.io/databases/part-4.html
This blog has moved. This is reposted from Paired Ends:
https://blog.stephenturner.us/p/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface
Llama 3.1 405B\u00a0is the first open-source LLM on par with frontier models GPT-4o and Claude 3.5 Sonnet. I've been running the\u00a070B model locally for a while now\u00a0using\u00a0Ollama\u00a0+\u00a0Open WebUI, but you're not going to run the 405B model on your MacBook.
Here I demonstrate how to create and deploy a Llama 3.1 405B-powered chatbot on any GitHub repo in <1 minute on HuggingFace Assistants, using an R package as an example
I'm going to use the\u00a0tfboot\u00a0R package as an example here (paper,\u00a0GitHub). I wrote the tfboot package to provide methods for bootstrapping transcription factor binding site disruption to statistically quantify the impact across gene sets of interest compared to an empirical null distribution. The package is meant to integrate with Bioconductor data structures and workflows on the front end, and Tidyverse-friendly tools on the back end. You can read more about the package in the\u00a0paper.
The 42-second video below demonstrates how to create & deploy your chatbot.
Once you create and activate your model, you'll see an interface that will look familiar if you've ever used ChatGPT or similar LLMs.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5fe747a-1a34-4be4-9421-325ba121a0dc_943x746.png\")
From here you can click one of the example prompts, or type your own prompt. Let's give it a try. First, a softball pitch. What does this package do? This should be fairly obvious from the README.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c782a3d-ebed-4a0c-9461-1880385f682a_736x898.png\")
Next, let's get a little basic info on usage. The chatbot looks through the package's RMarkdown vignettes and pulls out a high-level protocol on how to run the analysis. There wasn't much context on what motifbreakR is or what you have to do upstream of running tfboot, but further prompting can help with this.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd145222d-9647-4641-9ccd-4e44d55330f3_767x1109.png\")
Finally, let's see what it can tell us about the statistical underpinnings of what the package is doing? Of note, this isn't simply regurgitation of what's in the package documentation or vignettes. It's using a combination of the code and documentation itself and integrating that with general information about bootstrapping, null hypothesis significance testing, and transcription factor binding site disruption analysis.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ccabdb8-d3ab-469e-a3ed-87aaea01fea6_759x1286.png\")
Keep in mind that any assistant you create will be public. You can play around with the tfboot chatbot\u00a0here. Also, know that the 405B model is extremely resource intensive. A few times the bot would timeout and I'd have to retry the prompt. This happens far less often with the 70B model, and the response times are faster. You might experiment around and see for yourself where the speed/accuracy sweet spot is for your specific needs.
","doi":"https://doi.org/10.59350/5hh2d-tnq35","guid":"tag:blogger.com,1999:blog-6232819486261696035.post-5846799348349524841","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1726012800,"rid":"85ms1-zq218","summary":" This blog has moved. This is reposted from Paired Ends: https://blog.stephenturner.us/p/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface Llama 3.1 405B is the first open-source LLM on par with frontier models GPT-4o and Claude 3.5 Sonnet.","tags":["R"],"title":"Create a free Llama 3.1 405B-powered chatbot on any GitHub repo in 1 minute (cross-posted from Paired Ends)","updated_at":1788597378,"url":"https://gettinggeneticsdone.blogspot.com/2024/09/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface.html","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"This blog has moved. This is reposted from Paired Ends:
\n\n
PLANES provides a set of methods for evaluating the plausibility of epidemiological signals and forecasts. The PLANES methods are available in the rplanes R package and Shiny app.
Since early 2023, we've been developing methods to conduct what we are calling\u00a0plausibility analysis of epidemiological signals (PLANES). The motivation for PLANES stems from our work on infectious disease forecasting projects, including\u00a0COVID-19,\u00a0influenza-like illness (ILI), and\u00a0influenza hospitalizations. Near-term forecasts of disease patterns can help guide public health decision-making. We've been fortunate enough to participate in several consortia efforts that openly solicit contributions from modelers using different methods. Those methods are often tailored to use training data that comes from epidemiological reporting systems. The openness of these forecasting \"hubs\" is invaluable, and can lead to\u00a0ensemble forecasts\u00a0that as a whole are greater than the sum of their parts.
But after all we're trying to predict the future, so there are plenty of challenges! To name a few:
These particular challenges underscore a need for review of both the incoming surveillance data and forecasts that are generated. But that review requires human intuition \u2014 a \"gut feeling\" \u2014 and it can be hard to scale.
Enter PLANES. We developed this approach to reduce the burden of human review for epidemiological signals. The aim here is to mirror human intuition, but not replace it. In other words, by flagging\u00a0implausibility\u00a0we are not claiming\u00a0impossibility. Instead, the approach provides a screening mechanism so that humans can take a closer look at the signal data. PLANES is agnostic, such that it can be used across either forecasted or observed signals, for different diseases, and for varying temporal and geographic resolutions. The PLANES methods are described in detail in a manuscript we published in August 2024:
Nagraj VP, Benefield AE, Williams D, & Turner SD. (2024). PLANES: Plausibility Analysis of Epidemiological Signals.\u00a0medRxiv,\u00a010.1101/2024.08.22.24312449.
The concept behind the algorithm is demonstrated in Supplemental Figure A1 from the manuscript (below), which illustrates notional components and impacts.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58b2a85f-6645-49e8-ad87-2446bf82eaf3_1280x962.jpeg\")
Once we established the PLANES methods, we moved on to the\u00a0rplanes\u00a0R package to implement this approach.
A human can look at a time series of epidemiological data (e.g., incident influenza hospitalizations) and a probabilistic near-term forecast, and get a good sense of whether the forecast \"looks weird\" or not. PLANES attempts to systematize and formalize this assessment such that it can be automated and scaled. We created multiple components, each of which is a binary (yes/no) assessment of plausibility that maps to some feature in the data. All evaluated components are then combined into an ordinal score. By default, each component is equally weighted in the overall PLANES score. When delivered in the rplanes R package, the user can optionally weight components higher or lower in the scoring scheme.
The difference component checks the magnitude of point-to-point differences for all time steps of the evaluated data. This component can be used on either forecasts or observed signals. If an evaluated signal departs from the prior observation more dramatically than has been seen previously in the time series, then it is flagged as implausible. The function internally computes the maximum observed difference (based on absolute value) to set a threshold, which if exceeded will trigger a flag to be raised by the algorithm. While large and unexpected point-to-point changes may naturally occur in epidemiological signals, this component provides a means to draw attention to the most extreme cases.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0749393-f849-48f9-9931-7ee5aef65324_1280x884.jpeg\")
The coverage component compares the prediction interval for the first horizon of the evaluated signal to the most recent value in the seed. If the interval does not cover the most recent data point, then the flag is raised as implausible. The width of the interval used for this evaluation can be customized. The narrower the width of the prediction interval, the more sensitive this component will be.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12140766-26a7-4989-b7f9-b12f61f835d8_1280x884.jpeg\")
The taper component checks whether the prediction interval for the evaluated signal decreases in width (i.e., certainty increases) as horizons progress. The width of the prediction interval at every horizon is assessed against the previous horizon and if any of the intervals for the earlier horizon is wider a flag is raised. One would expect that there would be more variability in signals forecasted further out in time, and therefore the prediction interval would be wider in later horizons.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F622b4824-4a37-47e8-a0e6-e15095d0d0cf_1280x883.jpeg\")
The repeat component checks whether consecutive values in an observed or forecasted signal are repeated more than the tolerated number of times (k). This stores the maximum number of consecutive repeats for each location and uses this as the default value for\u00a0k. If the evaluated data exceeds\u00a0k, then the signal is considered implausible and a flag is raised.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b817e78-bdec-4425-bafd-58d39f46dfee_1280x883.jpeg\")
The trend component assesses if there is a significant change in the magnitude or direction of the slope for the evaluated signal compared to the most recent data in the seed. Each \"change point\" in the signal is identified using a hierarchical divisive algorithm originally implemented in the\u00a0ecp R package.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F31410b56-85b3-4656-be39-d72c94e231e9_1280x883.jpeg\")
While the trend component scans the time series for an inflection point, the shape component assesses the time series for unusual shapes across multiple points. To arrive at the shape assessment, the algorithm first divides the observed seed data into sliding windows to form trajectories. The trajectories are summarized as a set of shapes against which the forecasted trajectory is compared. If the shape of the forecasted trajectory does not match any shapes in the seed data, then the forecast is considered implausible per this component. The core intuition underlying this component is that the shape of future data is more likely to reflect patterns that have previously been observed and less likely to be a novel trajectory. Therefore, it may be useful to flag any novel shapes for review. The PLANES paper describes the two methods we used (dynamic time warping, and differences of consecutive observations) to summarize the shapes of signal trajectories.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c6d7930-65d1-42e1-8d36-e9d91f2c698b_1280x884.jpeg\")
The zero component was designed to check if there are any \"sudden\" zeros in the evaluated signal. Whether it is a broken surveillance instrument or miscalibrated forecast, we expect it would be unlikely to observe a zero if it has never been reported in the signal data.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5601502c-d93d-4cad-82f4-adcb6e6b4ce3_1280x883.jpeg\")
Now that we've covered all of the individual PLANES components, let's take a look at the rplanes R package, which implements functions and a Shiny app to evaluate epidemiological signals and forecasts for these components.
rplanes\u00a0is released under an open-source license, with code, documentation and issue tracking on GitHub:\u00a0https://github.com/signaturescience/rplanes.
The package is also released on CRAN:\u00a0https://cran.r-project.org/package=rplanes.
The package website includes function documentation, reproducible examples, and narrative vignettes describing how to get started as well as advanced usage of the tool:\u00a0https://signaturescience.github.io/rplanes/.
We tried to develop\u00a0rplanes\u00a0to be as intuitive as possible. Before assessing the individual PLANES components that we defined, we needed to ensure that user data was formatted consistently. We also needed a structure to store location-specific baseline characteristics against which we could compare the evaluated signal. With this in mind, we created an object-oriented structure using S3 classes in R. The figure below walks through the workflow for preparing data and analyzing data in\u00a0rplanes.
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86af6d40-05bc-4505-8338-075665c2d8bc_1257x1280.jpeg\")
The rplanes package has several\u00a0vignettes\u00a0illustrating\u00a0basic usage,\u00a0detailed descriptions of the PLANES components,\u00a0interpreting plausibility scores, and the\u00a0rplanes explorer shiny app.
In addition to a programmatic interface, we included a point-and-click version of the tool. This is developed as a Shiny app that is built into\u00a0rplanes. We've translated all of the package functionality to widgets in the app. To launch the app simply run\u00a0rplanes_explorer().
![\"\"](\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58ef04e2-5b18-4cfe-8629-73cb733ff580_3456x2234.gif\")
rplanes_explorer(). For details, see the\u00a0rplanes explorer vignette.
I first read about Lens.org via a tweet on my way back from a conference in April 2018. There seemed to be something in the water at the time, as they was an explosion of new discovery services and idexes in the past few months, including Digital science's Dimensions, 1Science's 1Findr, Scilit and the new resurgent challenge to Google Scholar posed by Microsoft Academic.
At the time, I took a quick look at Len's about page to get a better sense of what the data included.
I immediately noted that Lens had a strong focus on patent searching (it's predecessor Patent Lens was built in 1999 to \"render the global patent system more transparent\"), and article search was the area I was most interested in so I perhaps prematurely dismissed Lens as mainly Microsoft Academic Graph (MAG) + patent searching.
However I have come to realise that Lens might in fact be a far more exciting development than I thought.
While it is true that the scholarly search portion of Lens might be perhaps mostly dominated by the voluminous data from Microsoft Academic Graph (MAG), Lens is far more than the sum of it's parts by combining open data from half a dozen open data sources.
The other significant thing about Lens that differentiates it from the other search discovery and citation indexes is that it is run by Cambia a non-profit that seems committed to produce a open, free to use alternative to commerically owned and licensed indexes.
Most importantly, Lens is not just open data but there is a surprisingly feature rich interface that goes with it that can go toe to toe to many commerical interfaces. Given that the tool is still very new (Scholarly citations from Microsoft Academic Graph was added only 2 weeks ago) , this is pretty impressive and the speed of updates so far, makes me wonder how far it will go, though of course it is still early days.
This is going to be a detailed review in two parts. The first part will focus on the features offered by Lens, wth a focus on the Scholarly search portion of Lens.
This will be followed by a simple test of the data in Lens by extracting items with a given affiliation and comparing the overlap with data from Scopus/Scival.
A detailed analysis of the results will be provided in my next post.
Table of content
Lens is supported by Queensland University Technology and Cambia, which is self-described as \"a non-profit social enterprise that for over two decades has been dedicated to democratizing problem solving using science and technology.\"
I have never heard of Cambia until now, but I understand they have been in the patent search space (starting from life sciences before expanding to other patents) since the 90s. With funding from organizations like Rockefeller Foundation , Bill & Melinda Gates Foundation they have slowly expanded the capabilities of Patent Lens. pooling data from various Patent offices around the world, and added innovative features like Patent Sequence Search . I'm probably not doing full justice to the history of Cambia and Lens, please read the timeline for more details.
Lens makes the following pledge.
.
Open, free, privacy - pledges by Lens
Privacy conscious, Free (no premium options) and Data that is open ? Sounds too good to be true. The librarian in me wonders about sustainability and business models but as this post focus on the features of Lens, I shall refrain from commenting further.
The most interesting aspect of Lens to me besides the patent related functionality, is how it blends together data from various open data sources - including
This is very similar in concept to OpenAIRE's DOIBoost project which aims to enhance Crossref metatdata with MAG, ORCID and Unpaywall data but besides the difference in sources used, Lens has a full infrastructure around it and allows anyone to query and use without any technical expertise for free.
The idea here is that by combining all these open sources, one can cover for the gaps in any single data source to provide a super \"meta\" record that is better than any individual source.
Below shows one particularly complete record in the search results.
A particularly complete record in Lens search engine result page
Besides Unpaywall's links to open copies, the latest version of Lens even tries to use the OCLC registry to try to redirect you to your institution's resolver to access items via institutional subscriptions.
Unfortunately, my hunch is not that many libraries maintain their Openurl settings in WorldCat registry so this might not work for many people. Also the release notes that this feature currently works only for records with doi, though this might change in future versions.
Most of the tags listed such as \"open access\", \"patents\", \"affiliation\", \"Field of study\" , \"Clinical trial\" reflect data from various sources, I can hazard a guess where most of them come from.
Still, in case you are wondering, the \"Full text\" tag also indicates Lens has the full text of this paper indexed probably via CORE possibly the largest open repository of full text of Scholarly item.
Clicking into the record, you will see this impressive full record.
A particularly complete Lens record - Full record view
On the top right you can see various pid (permanent IDs) from Crossref, Microsoft Academic, Pubmed, Pubmed Central and a pid created by Len itself, so you can see what sources the combined record uses & click through to resolve the pids.
PIDs in the Lens record
To learn more about the Lens ID, refer to this help file , but it is essentially \"15 digits long, broken in five groups where every 3rd digit is separated from the next with a dash (-)m followed by a check digit.
Below that, you get various information, in particular citation counts from Scholarly material and patents, references in the paper, funding info, links to Open access version and more.
Citation metrics in Lens full record
In many ways this combined record compares well against commerical A&I records from Scopus, Web of Science and Dimensions.
Lens Full record is comparable to commerical alternatives
Of course, not all the records will be that complete, it all hinges on the completeness of the various sources Lens drawn from.
Lens default search box offers Patent (default) and article search
On the Lens homepage you can choose to search for patents or by articles (clicking on the \"Scholar\" button). If you just enter your query and hit enter, patents will be the default search.
Use the drop down arrow to do a structured (advanced field) search
Perhaps of interest to power users will be the \"Structured Search\" option, this is effectively the advanced search. The \"classification viewer\" option is for patent search.
Lens - structured field search for articles
The structured search will have a toggle to switch between Scholarly articles and patents.
Looking at the fields for Scholarly search can be awe inspiring, as you can see there is a wide variety of possible fields to search through , all thanks to the various data sources Lens pulls from.
Some fields you can use in structured field search
You get the usual title, abstract, author, keywords, pub type etc fields
Significantly you get \"Author affiliation Name\", so you can search by institutions, something you can't do easily with most free services (the exception is Microsoft Academic though you can't filter direct but need to requite on the semantic search - type and select from drop down) . I'll talk more about this later.
Restricting to affiliations in Microsoft Academic
You also get less commonly seen fields related to
Controlled vocabulory fields like MESH (from Pubmed), Field of Study (from Microsoft Academic)
Funding (eg Funding name, Funding Country, Funding ID etc)
Clinical trials (eg Clincial trial registry, Clincal trial ID)
Chemical substances (e.g Chemical Registry Name)
Reference (by Reference ID and Reference by Len Id)
Sources (e.g Source Country, Publisher, ISSN, Source subject etc)
Conference (e.g name, location, year etc)
Open Access (e.g Color, license)
I'm not too familar with some of these fields, but it's obvious some like the Open Access data is drawn from Unpaywall , a lot of STEM related fields are drawn from Pubmed, sources are probably from Crossref etc, while the more common ones are merged from more than one source.
In a way, the fact that it has a lot of filters is also a weakness. It isn't easy to tell how completely populated the fields are. For example, where is the Funding data drawn from? How complete are they? Does every record have a author affiliation field? That said there are flags you can use to restrict your search.
Flags in Lens
For example, you could restrict your search to only records with funding data, if they were open access, or even only if Lens had full text indexed.
Lens has full blown support for boolean operators which together with comprehensive filters, search history , saved queries and saved collections (compares to \"saved lists\" features for many databases) means Lens is feature rich enough even for power users. But in some sense this may not be as relevant as it seems as I wonder how many users will use Lens as a purely discovery tool when Google Scholar exists.
Search History in Lens
I would rate the emergence of Google Scholar as a scholarly discovery tool has perhaps the most significant development in this field so far in the last 20 years. While it is by no means perfect and lacks some features wanted by power users, it's existence and popularity is the reason why it's extremely difficult now to charge for a general purpose scholarly article discovery index. (My view is Scopus and Web of Science mainly persist due to their importance as citation indexes, while other subject specific A&I have specific disciplinary features).
This is reflected perhaps by the fact that one of the newest entries - Digital Science's Dimension provides a freemium discovery search that matches up with Google Scholar but adds the conventional filters that Google Scholar lacks and even throws in some visualizations of results.
Microsoft Academic is almost eeriely similar as a discovery tool. (To be fair both Dimensions and Microsoft Academic boast about using latest Machine learning and Natural language processing techniques for entity recognition and topic modelling, but we are talking about features users can use and it's arguably how much of what techniques actually lead to an improvement in relevancy of results compared to what Google Scholar is doing)So it is no surprise that Lens as a Scholarly discovery tool follows the same playbook when displaying the results and shows not only results, filters but also a visualization panel on the right.
Search result page in lens with filters and visualization panel
Still Lens is perhaps servicable as a discovery service if for some reason you want a change of pace from Google Scholar and use a tool that is designed for power users.
For sure the size of the index is not an issue, as of writing Lens has over 194 million records , which would cover the vast majority of peer reviewed material and Scholarly items you would want to find.
For comparison as at time of writing Microsoft Academic is boasting of 200 million records - which makes me wonder again about how much this one source is impacting Lens and whether the Scholarly search of Lens is just Microsoft Academic really.
That said searching Microsoft Academic and Lens with the same search keywords will give you very different results (far fewer in Microsoft Academic) because Microsoft Academic uses a \"semantic search\" rather than traditional keyword matching!
For further context on sizes of databases, Latest research as of 2018 estimates the sizes of the following databases
Google Scholar - 389 million
Proquest (selected 19 databases) - 279 million
Ebscohost (Sslected 375 database) - 132 million
Bielefeld Academic Search Engine (BASE) - 117 million
Web of Science (10 databases including core collection) - 105 million
Scopus - 72 million
Web of science (core collection) - 67 million
So clearly Lens is up there with the biggest , except for the might Google Scholar of course.
You could use Lens as a discovery tool by typing keywords but the fact that the default ordering is by Scholarly cites gives you a slight cue perhaps that this isn't the main purpose?
Still you can change the sorting to any of these
Relevancy
Scholarly citations
Patient citations
Publication date
Source Subject
Source type
Somewhat similar to Dimensions, you can see visualizations of the results set on the right of the screen. You can choose to open this as a full screen.
Full page visualization panel in Lens
By default you get institutions displayed as a logo grid as well as various visualizations of authors, journals, Field of Study, year of publication etc.
But you can visualize almost any of the other fields such as Funding, Open Access Color, Open Access License, Conference related fields, Citation type etc.
I was wondering what the \"citation type\" visualization does, but it basically shows you the source where the data comes from.
For instance take this visualization of the research set by searching for items with author affiliation for my institution (filtered from 2013-2017).
Citation type in Lens
As you can see, the result set has 2,345 records but magid (Microsoft Academic ID) alone contributes 2,329 records. Crossref (dois) are almost as numerous and there are very little from Pubmed (pmid) or Pubmed Central (pmcid) sources.
This makes a lot of sense since my institution does not have do research in life sciences. It also shows there is a huge overlap between Crossref records and MAG and in this case at least MAG \"knows of\" almost all the records in the result set, though no doubt the merger of both Crossref and MAG might provide better quality of data than just using MAG.
Of course, the real selling point about Lens is that it shows you the citation counts of result sets at various levels up to institutions.
Let's talk about that.
In many ways, tools like Scopus and Web of Science are valued today less for discovery per se but the citation metrics they provide. I suspect many libraries subscribe to them for this very reason.
To be fair Google also has metrics and you can find papers citing papers you are looking at but functionality is limited, though admittedly the author profiles in Google Scholar have had good takeup rates.
What Google Scholar and most freemium services like Dimensions lack is the ability to compare by institutions and to export in large batches.
Many institutions pay Elsevier and Clarivate large sums for Scopus/Scival and/or Web of Science/Incites for access to such features for benchmarking of institutions. Research Offices and libraries hunger for information on such benchmarks and measuring of impact .....
Can Len do some of this for free? Surprisingly yes.
Like any normal citation index, Lens allows you to find Scholarly citations of each paper by clicking on it. Until two weeks ago, this citation count included only open Crossref citations which is known to be lacking due to hold outs like Elsever, IEEE, ACS, but now it includes citations from Microsoft Academic Graph, which of course makes Lens citations far more complete, thiough perhaps at the cost of some accuracy as MAG data is mostly automatically extracted from bots.
Another piece of good news. Lens allows you to filter by affiliation, a feature usually not available in free versions. To do so, go to the structured search, select Scholarly toggle and use the author affiliation field.
Using structured search to search for items with affiliation
For my institution this gets me almost 6k records, you can add a bunch of filters after that, but the main trick is you can do the same visualization as before to study aspects of the outpost of your institution.
Visualization of institution output in Lens
For example you get handy stats for total cites from Scholarly material and patents to the almost 6k publications of my institutution. So for my set of results of 5,933 records
134 them were cited by patents
This came from 382 patents
There were 496 total patent citations
3,962 records were cited by Scholarly items
This came from 75,226 Scholarly items
With total cites of 90,855
Cites from patents rather than scholarly items is perhaps a different way of measuring impact of articles than the usual citations. Elsevier's Scival has a tab showing \"Economic impact\" which shows this very metric. I will talk more about this in the next blog post.
Besides citation metrics, you can also get the same visualizations as before, in particular you can see how much of the output is open access, by license, by top journal titles, collobrating institutions etc.
But besides showing useful statistics, Lens would be less useful as a tool, if one was unable to batch export the results for further study.
In fact, Lens is very capable here too. Once you have registered an account (possibly linked to Linkedin or ORCID) you can add all the results in a search and add to a collection.
Adding results to collections in Lens
Below shows some collections I have created. Some are for institutions for my comparisons and some are for authors.
My collections in Lens
Such collections can be set up to be shared publicly or privately and most importantly do a batch export up to 50k records in CSV, BibTeX, RIS, JSON formats
Exporting up to 50k records in Lens
The last is a particularly generous feature, given that Scopus only allows 20k exports. This should be sufficient for most bibliometric analysis of even the largest institutions (in a batch or two).
It is somewhat disappointing though that while you can export a bunch of fields , unlike Scopus or Web of Science you can't export the references, which can be useful for bibliometric analysis using tools like VOSviewer.
One workaround which doesn't quite scale for large datasets is to go to the full record of every item, click on the citations tab.
Then lower down, select \"XXX references\" and click on view articles in Scholar Search
On the resulting search result page, add to a new collection. Rinse and repeat as required, then do a batch export. This works also if you selected \"XXX Scholarly Citations\" instead.
Here's a tip, you can search directly for references of a an item in Lens by going to the structured search , choosing the field \"Reference ID\" and entering the Lens ID of the item (without hyphens).
Lens is clearly still a state of evolution, so what follows is just a preliminary assessment
Compared to expensive tools like Scival and Incites or Dimensions (for analytics), it lacks more sophisticated metrics like Relative Citation Ratio (RCR) , Field Weighted Citation (FWCI) and even the simple H-index etc though if you really wanted to, you can sort by citation count and manually get the H-index. I've found sorting by citations somewhat quirky and occasionally the sorting doesn't work properly, though reloading often works.
Lens currently still lacks an API but I'm told this will be coming soon.
Of course, Scival and Incites have much better features for benchmarking than Lens (e.g. the function of plotting various institutions against each other), but to be fair Lens should be matching up with Scopus/Web of Science rather than Scival/Incites and against them Lens is more than capable function wise.
That said, citation indexes need to be trusted to be accepted. As of now, there are only 2 citation indexes that are taken seriously - Scopus and Web of Science. Both of them are over a decade old and are used if (only partly) for many real world decisions in particularly University rankings and Faculty assessment.
Google Scholar is the third one often mentioned by faculty who like the higher citation counts, but IMHO is unlikely to be seriously accepted in say University rankings because the data is not transparent and cannot be easily studied (lack of infomation, batch exports and API) .
Digital Science's Dimensions is my bet to be the third widely accepted citation index (because they have made in roads by working with institutions) , though it will take years if not decades for this to occur.
But none of these are free, will Lens make the grade as a accepted citation index eventually?
The elephant in the room of course is I have been talking about features of Lens, without considering the quality of the data. Given that Microsoft Academic Graph is a big part of the data, one might suspect the quality of the data relies heavily on that source. One also wonders at how well the merging process occurs, for example which source takes priority if there is a conflict? How does the mapping occur between publication types listed by Crossref, PubMed and Microsoft Academic Graph?
I will cover in greater depth a couple of tests I did in my next blog post, but here are some preliminary results I got in a test I did by comparing items in Lens with author affiliation - \"Singapore Management University\" against the output I obtained from Scival.
Output was matched first using DOI , followed by a fuzzy match of titles ( matched only if pub year is less than or equal to a year).
As you can see the overlap is fairly respectable (52.5%). A slightly surprise was that the output of Lens for my institution (2,345) was only slightly bigger than Scival's (2,271).
A interesting study to compare against would be this study by Leiden University on \"Accuracy of affiliation information in Microsoft Academic\", though the study used Web of Science to compare not Scopus, given that Microsoft data is a big component of Lens,
I then analysed the differences in results, with led to quite interesting insights but that will have to wait for the next post.
OLMo 2 1B is a new generation of open-source language models launched by the Allen Institute for AI, with a parameter scale of about 1 billion. It provides excellent natural language understanding and generation capabilities while maintaining a small size, and is suitable for a variety of downstream tasks.
In this post, we will give a run-down of the complete installation process, fine-tuning, and evaluation of the OLMo-2-0425-1B model from AllenAI. We explore how to use LoRA (Low-Rank Adaptation) to fine-tune the model efficiently and evaluate its sentiment classification ability with a custom dataset, all running locally with MPS acceleration.
OLMo-2-0425-1B is a 1.1 billion parameter decoder-only language model released by AllenAI on May 1,2025. As part of the OLMo series, it is trained from scratch using high-quality, open-source datasets with a strong emphasis on transparency, reproducibility, and research accessibility.
Key characteristics of the model include:
OLMo-2-0425-1B is not instruction-tuned or chat-optimised by default. Instead, it is designed as a research-friendly base model\u2014ideal for experimenting with new downstream tasks such as classification, summarisation, or dialogue systems. So it is easily tested on a local computer because:
conda create -n olmo-fresh python=3.10 -y\nconda activate olmo-fresh\npip install torch torchvision torchaudio\npip install transformers datasets peft gradio\nAnd if you keep meeting the problem like this
RuntimeError: operator torchvision::nms does not exist
or
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
It means you may have manually installed torch / torchvision from different sources or multiple libraries loaded different versions of the OpenMP (libomp) dynamic library, causing conflicts. We recommend that you uninstall torch/torchvision or create a new virtual environment. This is the quickest way to solve these kinds of problems.
OLMo 2 1B is supported in transformers v4.48 or higher:
\npip install transformers>=4.48
You can use OLMo with the standard HuggingFace transformers library:
\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai/OLMo-2-0425-1B\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai/OLMo-2-0425-1B\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\nAfter running the script above you will see the output like this:
\nLanguage modeling is a type of language learning methodology that is concerned with acquiring linguistic competence without having the necessary raw input in the target language (L2).
\nNatural language processing (NLP) is the study of techniques for computers to process, understand, and manipulate human languages. Natural language processing is a broad discipline that also focuses on aspects of language acquisition.
\nTranslation models is a type of software program used in translation to translate text from one language to another. The most popular translation model is statistical
\n`
As we mentioned before OLMo-2-0425-1B is designed as a research-friendly base model, so we want to test a new task Sentiment Classification on OLMo-2-0425-1B. We want to classify the sentiment contained in a given sentence as positive or negative. Here is an example
\nInput sentence:
\n\"I really hated the service at this restaurant.\"
Output:
\nNegative
Here is the source code : Git repository
\nFine-tuning is the process of adapting a pretrained language model to a specific downstream task\u2014such as sentiment classification\u2014by continuing its training on a smaller, task-specific dataset. Instead of training a model from scratch (which is computationally expensive and data-hungry), fine-tuning allows us to leverage the general linguistic knowledge already captured by large-scale pretrained models.
\nHowever, full fine-tuning of large language models (LLMs) like OLMo-2-1B can be resource-intensive, especially when running on consumer hardware.
\nLoRA (Low-Rank Adaptation of Large Language Models) offers a solution to this challenge. It is a parameter-efficient fine-tuning technique that:
\nIn our case, LoRA allowed us to fine-tune OLMo 2 1B on a MacBook Pro using MPS acceleration without running out of memory.
\nHere's a simplified visual of how LoRA operates within a transformer layer:
\nW (frozen)\n + A @ B \u2190 (trainable low-rank adapters)\n ---------\n \u2193\n Modified Attention Output\nWhere:
\nWe use GPT-4o to generate a small synthetic sentiment classification dataset, for each sentence we give it a label, here is the one example data in the dataset:
\n{\n \"text\": \"I absolutely loved the product.\",\n \"label\": \"positive\"\n}\ntrain.json(160 data) and val.json (40 data)for training and validation.We used PEFT to apply LoRA to the OLMo model and fine-tuned it on our sentiment dataset. Here is my full project path:
\nOlmo/\n\u251c\u2500\u2500 data/\n\u2502 \u251c\u2500\u2500 full.json # full dataset\n\u2502 \u251c\u2500\u2500 train.json # training subset\n\u2502 \u251c\u2500\u2500 val.json # validation subset\n\u251c\u2500\u2500 train.py # main LoRA fine-tuning script\n\u251c\u2500\u2500 infer.py # inference script\n\u251c\u2500\u2500 test.py # accuracy evaluation\n\u251c\u2500\u2500 train_loss_log.json # training loss record (JSON)\n\u251c\u2500\u2500 split_data.py # utility to split full.json\n\u251c\u2500\u2500 visualize.py # loss plotting script\n\u251c\u2500\u2500 olmo-lora-adapter/ # saved LoRA adapter\n\u251c\u2500\u2500 checkpoints/ # chech point save dir\npeft_config = LoraConfig(\n r=8,\n lora_alpha=16,\n task_type=TaskType.CAUSAL_LM,\n lora_dropout=0.1,\n bias=\"none\",\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]\n)\ntraining_args = TrainingArguments(\n output_dir=\"./checkpoints\",\n per_device_train_batch_size=1,\n per_device_eval_batch_size=1,\n eval_strategy=\"epoch\",\n save_strategy=\"epoch\",\n num_train_epochs=10,\n learning_rate=2e-4,\n logging_steps=10,\n report_to=\"none\",\n fp16=False\n)\nWe also defined a LossLoggerCallback to track and save loss during training:
[\n {\"step\": 10, \"loss\": 12.3},\n {\"step\": 20, \"loss\": 9.1},\n ...\n]\nSaved as train_loss_log.json. And here is the loss curve
\n
We defined an infer.py script to evaluate single sentences:
prompt = f\"<|user|>\\nClassify the sentiment of this sentence: \\\"{sentence}\\\"\\n<|assistant|>\\n\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(device)\noutputs = model.generate(**inputs, max_new_tokens=5)\nPredicted label is extracted from the generated output.Here is one example:
\n
An evaluation script was used to compute classification accuracy on the val.json:
correct = 0\nfor example in val_set:\n predicted = classify(example[\"text\"])\n if predicted == example[\"label\"]:\n correct += 1\naccuracy = correct / len(val_set)\nWe find the accuracy is 1 and here is the confusion matrix:
\n
\nThe model was able to correctly classify a large portion of examples after just 10 epochs.
Through this experiment, we tested the real-world usability of OLMo-2-0425-1B, a 1.1B-parameter language model developed by AllenAI, in a local fine-tuning and inference setting. Rather than evaluating it solely as a pretrained base model, we explored how well it could adapt to a specific downstream task\u2014sentiment classification\u2014under resource-limited conditions.
\nOur results show that OLMo is a practical and flexible foundation model. With the help of parameter-efficient fine-tuning (LoRA), we were able to achieve reasonable performance using a custom dataset, all on a MacBook Pro without GPU. The model converged quickly, produced intelligible outputs, and showed a steady improvement in validation loss and classification accuracy over multiple epochs.
\nHowever, the testing also revealed some limitations. While OLMo handles structured prompts well, its output format can sometimes drift without proper formatting or prompt engineering. Additionally, compared to larger instruction-tuned models, it may require more task-specific guidance to maintain robustness in edge cases.
\nOverall, OLMo-2-0425-1B proved to be both efficient and adaptable, making it a strong candidate for researchers or developers looking to experiment with open models locally. Our testing confirms that even without cloud-scale hardware, OLMo can deliver meaningful results when paired with the right tools and training strategies.
","doi":"https://doi.org/10.59350/n8ant-1630","guid":"https://doi.org/10.59350/n8ant-1630","image":"https://hub.researchgraph.org/content/images/2025/05/olmo-logo-2.jpg","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1747267200,"rid":"1nsx6-ztq20","summary":"1 Overview OLMo 2 1B is a new generation of open-source language models launched by the Allen Institute for AI, with a parameter scale of about 1 billion. It provides excellent natural language understanding and generation capabilities while maintaining a small size, and is suitable for a variety of downstream tasks.","tags":["Artificial Intelligence","Toc"],"title":"From Zero to Classifier: Building a Lightweight LLM with OLMo 2 on Your Laptop","updated_at":1788597778,"url":"https://hub.researchgraph.org/from-zero-to-classifier-building-a-lightweight-llm-with-olmo-2-on-your-laptop/","version":"v1"},{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"Reposted from Paired Ends at\u00a0https://blog.stephenturner.us/p/llm-translate-documentation.
---
Most of the developer tooling for AI/LLM training and evaluation is Python-centric, but just over the past few months we've seen a surge of new tooling for AI/LLM\u00a0applications\u00a0for the R ecosystem.
ollamar\u00a0and\u00a0rollama\u00a0provide wrappers around the Ollama API allowing you to run LLMs locally on your machine. I recently wrote a few posts, one demonstrating how to use ollamar, and another demonstrating a package that uses ollama internally.
elmer\u00a0is a new package in the tidyverse that allows you to interact with many different LLMs via R (Claude, ChatGPT, Gemini, and Ollama too.
mall\u00a0is an interesting one that provides an easy way to run multiple LLM predictions against a data frame (sentiment analysis, summarization, classification, extraction, translation, etc).
pal\u00a0provides LLM assistants for doing things like highlighting code and asking for things like roxygen documentation, testthat tests, etc.
Shiny Assistant\u00a0helps you explain how things work in Shiny, and can help build Shiny apps for you (in either R or Python).
The\u00a0lang\u00a0package (source,\u00a0documentation) is an interesting new addition to the\u00a0mlverse\u00a0in R. From the documentation:
lang\u00a0overrides the\u00a0?\u00a0and\u00a0help()\u00a0functions in your R session. If you are using RStudio or Positron, the translated help page will appear in the usual help pane.If you are a package developer,\u00a0
lang\u00a0helps you translate your documentation, and to include it as part of your package.\u00a0lang\u00a0will use the same\u00a0?\u00a0override to display your translated help documents.
Let's look at an example. I recently invited my colleague and co-author VP Nagraj to write about the rplanes package we\u00a0published\u00a0and\u00a0released on CRAN\u00a0for plausibility analysis in epidemiological forecasting.
One of the first functions you might use from this package is\u00a0read_forecast(), which reads a probabilistic quantile forecast CSV file for downstream plausibility analysis. Let's look at the help for this function.
library(rplanes)\n?read_forecastNow let's get help in Spanish.1\u00a0load the lang package and tell it that we're using\u00a0llama3.2.2\u00a0We'll set the system language to Spanish, then ask for help again.
Sys.setenv(LANGUAGE=\"spanish\")\n?read_forecastMy fluency in Spanish is limited to general conversation and travel needs so I can't easily verify the accuracy of the translation of this technical language, but when I ran some of this back through Google Translate it seemed to be mostly faithful. Notice how things that shouldn't be translated aren't \u2014 function names, arguments, columns in the returned output, code in the examples.
What about non-Western languages?
Let's try Hindi!
Sys.setenv(LANGUAGE=\"hindi\")\n?read_forecastI can't verify the accuracy of this translation beyond running some of the text back through Google Translate, but in doing so at first glance the translation isn't bad.
What about Arabic?
Sys.setenv(LANGUAGE=\"arabic\")\n?read_forecastIf you're a native speaker of any of these, I'd love to know what you think. Chat with\u00a0me on Bluesky (@stephenturner.us).
The\u00a0lang documentation\u00a0has a great section on using lang as a package developer. You can translate all of your Roxygen documentation into the desired language, then edit those translations by hand as needed. Then a special helper function re-roxygenizes your docs placing them in a special\u00a0inst/man-lang\u00a0folder. The\u00a0lang docs\u00a0explain how this all works, but once you do this, when a user has the lang package loaded, they'll get your pre-computed and optionally edited translations instead of having to wait around for the LLM to translate the help.
Here's a demo using a very small package I wrote for something completely different. Don't worry about all the Docker stuff described here. There's one single function,\u00a0missyelliot(), that simply reverse complements a DNA sequence (\"take that flip it and reverse it\"). That is, it'll convert GATTACA to it's reverse complement TGTAATC.
Restart your R environment, and install the package using devtools/remotes. Load both rpdd and lang.
devtools::install_github(\"stephenturner/rpdd\")\nlibrary(rpdd)\nlibrary(lang)Now get some help for\u00a0missyelliot(). If your language environment variable is English, you'll get the English help.
Now, change your system language to Spanish, and try it again. Notice how the translated help is instantaneous \u2014 you're relying on the pre-translated and possibly hand-edited translations that come with the package rather than asking an LLM to translate the help for you on the fly.
Sys.setenv(LANGUAGE = \"spanish\")\n?missyelliotIf your language is set to something without a pre-populated translation, you'll have to register a model through Ollama and translate in real time.
Sys.setenv(LANGUAGE = \"russian\")\n?missyelliotReposted from the original at\u00a0https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input.
---\u00a0
If you use ChatGPT, Claude, or even some local model through\u00a0Ollama\u00a0or\u00a0HuggingFace Assistants, you'll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo. Here I'll demonstrate how to turn an entire GitHub repo into a single text file for LLM-friendly input, with R and Python packages as examples.
This\u00a0GitHub Repo to Text Converter\u00a0can help here.
Web app:\u00a0repo2txt.simplebasedomain.com
Source:\u00a0github.com/abinthomasonline/repo2txt
This little app takes a URL to a GitHub repo, lets you select which files in the repo directory structure to include, and creates a single plain text file you can feed into an LLM to ask for explanations, usage, or anything else you'd like to ask about the codebase. You can download or copy this text to the clipboard to later paste into an LLM of your choosing.
The tool runs entirely in the browser. You can also provide a personal access token if you want to access private repositories (again, securely, since everything runs in the browser).
I recently\u00a0demonstrated writing a CLI app using Click from a Cookiecutter template\u00a0in Python. It's a simple app that tells you how much caffeine remains in your system based on how much you consume and when you'll go to bed. It's on GitHub at\u00a0stephenturner/caffeinated.
Here I'm pasting the repo URL,\u00a0https://github.com/stephenturner/caffeinated\u00a0into the app. The tool is smart enough to recognize that this is a Python package, and I probably want the\u00a0__init__.py,\u00a0__main__.py, other package\u00a0.py\u00a0files, and the unit tests in\u00a0tests/\u00a0that I'll use with\u00a0pytest.
After hitting \"Generate Text File\" you'll see that it generates a single plain text file, first listing out the directory structure of the selected files, then listing out the contents of each file separately. Here's a preview. My\u00a0__init__.py\u00a0is empty (just to signal that this is a package), and the\u00a0cli.py\u00a0file is truncated here.
Directory Structure:\n\n\u2514\u2500\u2500 ./\n \u251c\u2500\u2500 caffeinated\n \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u251c\u2500\u2500 __main__.py\n \u2502 \u2514\u2500\u2500 cli.py\n \u2514\u2500\u2500 tests\n \u2514\u2500\u2500 test_caffeinated.py\n\n\n---\nFile: /caffeinated/__init__.py\n---\n\n\n---\nFile: /caffeinated/__main__.py\n---\n\nfrom .cli import caffeinated\n\nif __name__ == \"__main__\":\n caffeinated()\n\n\n---\nFile: /caffeinated/cli.py\n---\n\nimport click\nimport math\nfrom datetime import datetime\nfrom importlib.metadata import version, PackageNotFoundError\n\n### TRUNCATED ###Once you have this, you can easily paste this into an LLM of your choice to chat with the codebase. Here, I pasted this into GPT-4o with the leading prompt:\u00a0\"Explain this code to me, how to use it what it does.\"\u00a0GPT-4o returned all of the text and code below as a single response.
First, the directory structure.
Next, an explanation of\u00a0__main__.py,\u00a0cli.py, and the tests.
Next, a demonstration of how to use it. Note above that I did\u00a0not\u00a0select the README to add to my text file. GPT-4o divined this usage information from the code itself, rather than regurgitating the README.
And finally, a simple explanation of the (minimal) test suite.
This also works on an R package. In my last job I wrote a package to assist with containerizing R packages with a usethis-like interface. It's called pracpac (practical R packaging [with Docker]), and you can read more in the\u00a0paper\u00a0or on the\u00a0GitHub repo. Here I'll take the code from\u00a0pracpac repo\u00a0and pull out the relevant R source files.
This time I asked Claude 3.5 Sonnet to tell me more about this package.
If you look at the\u00a0source code\u00a0you'll see that this is a simple implementation using just HTML and JavaScript. Which means you can download the repo and open the index.html, or if you'd prefer a vanity URL, you can fork the repo and set up GitHub pages to serve from your main branch (e.g.\u00a0stephenturner.github.io/repo2txt).
","doi":"https://doi.org/10.59350/3t3wg-g6750","guid":"tag:blogger.com,1999:blog-6232819486261696035.post-352146836295256074","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1733702400,"rid":"qd318-eke39","summary":" Reposted from the original at https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input. --- If you use ChatGPT, Claude, or even some local model through Ollama or HuggingFace Assistants, you'll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo.","tags":["R"],"title":"Turn a GitHub repo into a single text file for LLM-friendly input (repost)","updated_at":1788597424,"url":"https://gettinggeneticsdone.blogspot.com/2024/12/github-repo-to-text-for-llm-input.html","version":"v1"},{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"Reposted from the original at\u00a0https://blog.stephenturner.us/p/python-cli-click-cookiecutter.\u00a0
---
In the spirit of\u00a0Learning in Public, I wanted an excuse to explore (1)\u00a0click\u00a0for creating command line interfaces, (2)\u00a0Cookiecutter\u00a0project templates, and (3) modern tools in the Python packaging ecosystem. If you're primarily an R developer like me, I recently wrote about resources for getting better at Python for R users.
Click\u00a0is a really nice package for creating command line interfaces, and I like it better than argparse or other similar utilities. Simon Willison's\u00a0click app cookiecutter template\u00a0was really helpful in getting the boilerplate set up for a python package, and while I looked at build backends like Flit, Poetry, uv, etc., I ended up just using setuptools with a pyproject.toml.
For this demo I built a silly little Python tool called\u00a0caffeinated\u00a0(inspired by\u00a0coffee-o-clock) that tells you how much caffeine you'll still have in your system at bedtime based on how much you consume and when. You can install it from\u00a0PyPI\u00a0and running\u00a0caffeinated\u00a0with the\u00a0--help\u00a0option (or without any arguments) prints the help. The code is on GitHub (github.com/stephenturner/caffeinated) if you want to follow along.
pip install caffeinated, then run\u00a0caffeinated --help\u00a0for usage info. Run\u00a0caffeinated -c 200 -b 9pm\u00a0to see how much caffeine will remain in your system if you consume 200mg caffeine right now and go to bed at 9pm.I'm not totally sure how accurate the formula is here, but I'm using this to calculate how much caffeine remains in circulation (and I'm going with 90mg for \"a cup of coffee\").1
Where:
N(t)\u00a0= Quantity of caffeine remaining
N0\u00a0= Original amount of caffeine
t\u00a0= Time
t6\u00a0= Coffee's half-life (6 hours)
Click (Command Line Interface Creation Kit) is a Python package for creating command line interfaces in a composable way with as little code as necessary.
Why Click? Why not argparse/docopt/etc?\u00a0Good questions. The Click documentation has a section on\u00a0Why Click?,\u00a0Why not Argparse?\u00a0and\u00a0Why not Docopt etc.?\u00a0I like click because it enables you to easily create command line utilities with subcommands (e.g.,\u00a0mycommand subcommand ...\u00a0e.g. like\u00a0bedtools intersect ...), and it supports file handling, makes it easier to handle options versus arguments, and easily supports ANSI coloring of the output.
First let's set up a folder structure we'll use to create a Python package. Make a new folder named whatever you're calling the package (in this case,\u00a0caffeinated), and in that directory create a\u00a0pyproject.toml\u00a0file. Create a new subfolder with the same name as the parent directory (caffeinated), and in that folder you'll have three files. Directory structure should look like this:
caffeinated/
caffeinated/
__init__.py
__main__.py
cli.py
pyproject.toml
The pyproject.toml will have just the basics you need for a Python package:
[project]\nname = \"caffeinated\"\nversion = \"0.1.1\"\ndependencies = [\"click\"]\n\n[project.scripts]\ncaffeinated = \"caffeinated.cli:caffeinated\"The\u00a0__init__.py\u00a0will be empty, and the\u00a0__main__.py\u00a0will just have one line that imports the function from the cli.py:
from .cli import caffeinated\n\nif __name__ == \"__main__\":\n caffeinated()The\u00a0cli.py\u00a0actually has the code for your command line tool. This is a really simple program that just echos out the amount of caffeine you consumed and what time your bedtime is:
import click\n\n@click.command()\n@click.option(\"-c\", \"--caffeine\", default=100)\n@click.option(\"-b\", \"--bedtime\", default=2100)\ndef caffeinated(caffeine, bedtime):\n click.echo(f\"Caffeine consumed: {caffeine} mg\")\n click.echo(f\"Your bedtime is: {bedtime}\")Now, pip install the package you just wrote:
pip install .And now the caffeinated command line utility is ready to use. First, get some help. Notice how by specifying a default value
$ caffeinated --help\nUsage: caffeinated [OPTIONS]\n\nOptions:\n -c, --caffeine INTEGER\n -b, --bedtime INTEGER\n --help Show this message and exit.Now run it:
$ caffeinated -c 200 -b 2100\nCaffeine consumed: 200 mg\nYour bedtime is: 2100You can see the real cookiecutter code here:\u00a0github.com/stephenturner/caffeinated. Here are links to the actual working code. Everything important is in the\u00a0cli.py\u00a0file. It adds a few more arguments, picks up the version from the pyproject.toml, and defines functions to do all the calculation and conveniences such as translating \"9pm\" into 2100 (hours).
Once you update all the source or just\u00a0pip install caffeinated\u00a0again from PyPI, the tool will tell you approximately how much caffeine you'll have remaining in your system after consuming a certain amount of caffeine at your chosen bedtime. Run\u00a0caffeinated --help\u00a0to get help on the options.
Organizing your project as a well-structured Python package can streamline development and distribution. The\u00a0Cookiecutter\u00a0package provides a straightforward way to generate project templates, ensuring consistency and best practices across your projects. Install it with pip, then I'll use\u00a0Simon Willison's click-app cookiecutter template.
pip install cookiecutter\ncookiecutter gh:simonw/click-appYou can see what this looks after running that at\u00a0this demo. Running this and answering a few of the prompts will create:
The directory structure described above
The\u00a0__init__.py,\u00a0__main__.py, and\u00a0cli.py\u00a0files with some boilerplate to get started.
A pyproject.toml file based on your answers to the prompts.
A tests directory with boilerplate for writing tests with pytest.
A README with badges pointing to a future PyPI release, changelog from your GitHub releases, license, and test status.
GitHub actions for publishing your tool as a package to PyPI (requires additional configuration\u00a0as described here).
The final caffeinated code made from Simon's click-app cookiecutter template is here\u00a0github.com/stephenturner/caffeinated.
I've written scores of R packages for fun and profit. There's really only one build backend toolchain for R packages that everyone uses:\u00a0devtools\u00a0with\u00a0Roxygen\u00a0documentation with liberal assistance from\u00a0usethis.
The Python documentation has a good guide on\u00a0Packaging Python Projects. The build backend ecosystem in Python is more diverse. I really wanted to take a closer look at\u00a0Flit,\u00a0Poetry,\u00a0Hatch, and others, but because the cookiecutter template that I used created a pyproject.toml using\u00a0setuptools\u00a0by default, so I just ran with that.
setuptools\u00a0is probably the oldest and most widely used packaging tools in Python with good documentation and community support. And with\u00a0PEP 517, the standard became using a simpler pyproject.toml rather than the old setup.py. You can see my\u00a0pyproject.toml for caffeinated here. It's pretty simple, and one key feature is the\u00a0readme=\"README.md\"\u00a0entry, which results in the documentation on the PyPI landing page (pypi.org/project/caffeinated/) populated with the README.md in the project root, avoiding the need for duplication.
Flit\u00a0(flit.pypa.io) looks like a very minimal, very simple build backend for packaging plain Python code. I also took a look at\u00a0Poetry\u00a0(python-poetry.org), because if I were building something more complex I think I'd want something to help me manage dependencies instead of having to add them to the pyproject.toml by hand. Poetry helps with this. Finally, there's a lot of interest in\u00a0uv\u00a0(docs.astral.sh/uv) right now. It's a Python package and project manager written in Rust, and the benchmarks are impressive. See the\u00a0\"uv: Unified Python packaging\"\u00a0blog post for more. uv doesn't yet have a build backend, but that's in the works at\u00a0astral-sh/uv#3957.
Building python packages with setuptools vs Flit vs Poetry vs uv might be the subject of a future post, but for now, I'm just using setuptools+build.
The pyproject.toml created by the cookiecutter template I used had just about everything I needed to build the package. From here it was simple. This will build the\u00a0.whl\u00a0binary file and\u00a0.tar.gz\u00a0source packages in a\u00a0dist/\u00a0folder.
python -m buildAfter this it's fairly straightforward to upload this to PyPI. But, before uploading to the production\u00a0pypi.org, you should probably upload to the testing repository (test.pypi.org) first to avoid polluting PyPI with broken or testing packages.
twine upload -r testpypi dist/*Uploading to the real PyPI follows the same convention. I recommend using a\u00a0token in your\u00a0.pypirc\u00a0file\u00a0instead of a username/password prompt.
twine upload dist/*A few seconds later your package will be on PyPI, and you can install it with pip install like you would any other package. The caffeinated package is on PyPI at\u00a0https://pypi.org/project/caffeinated/.
If you've ever tried getting a package into CRAN you know how onerous the process can be and how strict the CRAN maintainers can be2\u00a0and you may be shocked to see how easy it is to get a package onto PyPI. There is no curation or review process with PyPI. Upload your source and wheel files with twine and your package is live.
Once the app is on PyPI it's easy to create a Docker container. There's a Dockerfile in the repo that looks like this:
FROM python:3.11-alpine\nRUN pip install caffeinated\nWORKDIR /files\nENTRYPOINT [\"caffeinated\"]\nCMD [\"--help\"]You can build it like this (replace stephenturner with your Docker username):
docker build -t stephenturner/caffeinatedAnd push it up to Docker Hub:
docker push stephenturner/caffeinated This container is on Docker hub at\u00a0stephenturner/caffeinated. Now you can run it:
$ docker run stephenturner/caffeinated -c 200 -b 9pm\n\nYou would have 35.4mg of caffeine in your system if you went to bed at 9:00pm (in 15.0 hours).\nThat's like having 39% of a cup of coffee before bed.Alternatively, you can easily create an image with your new tool and additional tools you might want in the same container using\u00a0Seqera containers\u00a0(see the video linked further below for more details). I created\u00a0this image\u00a0that includes both caffeinated and\u00a0cowsay, and I'm piping the result of caffeinated into cowsay.
$ docker run --rm community.wave.seqera.io/library/pip_caffeinated_python-cowsay:5a33eb2abfe4e6a5 sh -c 'caffeinated --caffeine 200 --bedtime 9pm --start-time 8am | cowsay'\n __________________________________________ \n/ You would have 44.5mg of caffeine in \\\n| your system if you went to bed at 9:00pm |\n| (in 13.0 hours). That's like having 49% |\n\\ of a cup of coffee before bed. /\n ------------------------------------------ \n \\ ^__^\n \\ (oo)\\_______\n (__)\\ )\\/\\\n ||----w |\n || ||This is part of my TIL / Learning in Public series, which I wrote about recently:
For a quick intro on Click, see the official Click intro video. It's over 10 years old at this point, it's mostly still applicable.
This video from NeuralNine demonstrates how to create Click groups to create CLI utilities with subcommands:
This video from ArjanCodes explains how to create a Python Package and publish on PyPI. It uses the old setup.py instead of the more modern pyproject.toml convention, but otherwise it's still good to understand the steps in the process.
Finally, a little more about easily creating containers with multiple tools for multiple architectures using\u00a0Seqera Containers:
\u00a0This is reposted from the original at\u00a0https://blog.stephenturner.us/p/python-for-r-users.
---
A Google search for \"R vs Python\" returns thousands of hits across sites like Reddit, IBM, Datacamp, Coursera, Kaggle, and many others. A quick Google Trends analysis shows that this search query has grown steadily over the last decade.
Any real data scientist would agree that this argument is silly, that the right answer is to use the best tool for the job. What's \"best\" isn't always easy to answer. Sometimes the \"best tool\" might be the one that you're most familiar with and can write most easily. After all, compute time is cheap relative to software engineer time. In other cases there truly is a better choice depending on the problem (e.g., the AI ecosystem is generally more well developed in Python, while the Bioconductor ecosystem provides R with a rich framework for many kinds of bioinformatics tasks).\u00a0I'm partial to R/tidyverse for tabular data analysis and manipulation, but if polars or pandas is your thing, that's cool too.
Beyond \"choose the right tool,\" my biggest advice to trainees and students when asked about \"R vs Python\" was to learn both languages,\u00a0but learn one of them so well as to be native-level\u00a0fluent\u00a0in the language. That is, become so comfortable in a language that you rarely have to look up documentation, where you have a natural intuition about how to best implement a variety of your everyday tasks, where you know how to write code, docs, and unit tests without really thinking about it.
R is that language for me. I can read, write, debug, and package code in Python, but it doesn't come to me as easily. I've been collecting and bookmarking resources over the years that I've turned to for writing and reading Python, coming from native R fluency. LLMs have a new place in this as well, and I touch on that in the end.
If you're just getting started with Python or programming in general, there are thousands of courses and resources to help you get started. My favorite one to recommend is David Malan's CS50 course at Harvard. The course materials are\u00a0here, and the entire 16-hour course is available on YouTube. I still revisit the OOP section from time to time. It likely covers concepts you already know well, as this is meant to be an introduction to programming, using Python.
https://www.youtube.com/watch?v=nLRL_NcnK-4
learnxinyminutes.com has a collection of very short primers on different programming languages. These assume you've programmed in other languages before and understand flow control, OOP, functions, classes, etc. The\u00a0Learn X in Y minutes where X=Python\u00a0tour is here:\u00a0learnxinyminutes.com/docs/python/. It's a good place to start if you don't need to learn what a for loop is or how flow control works. This is simply a long Python scripts with comments throughout that demonstrate main features of the language for programmers coming from another language.
Reticulate\u00a0is an R package that enables interoperability between R and Python. It allows you to call Python from R, translate between R and python objects, and create bindings to different Python virtual environments or Conda environments.
The\u00a0Primer on Python for R Users vignette\u00a0in the reticulate package is a great place to start if you're a seasoned R user and want to know how to write Python code to do things in R that you're familiar with. The primer covers data structures, iteration, indexing, function definition, classes, dunder methods, iterators, generators, imports and modules, decorators, and more.
I've never personally met Emily but I've been following her since I saw her rstudio::global 2020 talk on building an R package ecosystem internal to your organization (video,\u00a0slides,\u00a0post). It's a great talk whether you're programming in R, Python, Rust, whatever, and building off of open-source tools packages for internal organization use only.
Emily recently published a three part series on Python ergonomics and patterns for seasoned R developers. Links to her posts with her description below.
Python Rgonomics: Switching languages is about switching mindsets - not just syntax. New developments in python data science toolings, like polars and seaborn's object interface, can capture the 'feel' that converts from R/tidyverse love while opening the door to truly pythonic workflows.
polars' Rgonomic Patterns: In this follow-up post to Python Rgonomics, we deep dive into some of the advanced data wrangling functionality in python's\u00a0polars\u00a0package to see how it's powertools like column selectors and nested data structures mirror the best of\u00a0dplyr\u00a0and\u00a0tidyr's expressive and concise syntax
Base Python Rgonomic Patterns: Getting comfortable in a new language is more than the packages you use. Syntactic sugar in base python increases the efficiency, and aesthetics of python code in ways that R users may enjoy in packages like\u00a0glue\u00a0and\u00a0purrr. This post collects a miscellaneous grab bag of tools for wrangling, formatting (f-strings), repeating (list comprehensions), faking data, and saving objects (pickle).
Additionally, I wasn't able to make it to posit::conf(2024) this year, but from a few colleagues who did, I heard her talk on the topic was great. The talks from the conference aren't yet online, but Posit usually posts them to YouTube a few months after the conference. Description of her talk from the\u00a0conference agenda:
Python Rgonomics
Data science languages are increasingly interoperable with advances like Arrow, Quarto, and Posit Connect. But data scientists are not. Learning the basic syntax of a new language is easy, but relearning the ergonomics that help us be hyperproductive is hard. In this talk, I will explore the influential ergonomics of R's tidyverse. Next, I will recommend a curated stack that mirrors these ergonomics while also being genuinely truly pythonic. In particular, we will explore packages (polars, seaborn objects, greattables), frameworks (Shiny, Quarto), dev tools (pyenv, ruff, and pdm), and IDEs (VS Code extensions). The audience should leave feeling inspired to try python while benefiting from their current knowledge and expertise.
Hadley, Mine, and Garrett's book R for Data Science is a must-read for anyone looking to upskill their R chops for data analysis and visualization, and the second edition is available to read free online at\u00a0r4ds.hadley.nz.
Arthur Turrell\u00a0and others have written\u00a0Python for Data Science\u00a0(aeturrell.github.io/python4DS), heavily inspired by R4DS, with most of the chapters mirroring R4DS. The image below shows the TOCs from both books. The book mostly uses pandas and lets-plot (\"ggplot2 for python\").
The Shiny for Python docs (shiny.posit.co/py/docs) are pretty good for getting into Shiny for Python, especially if you have some experience with Shiny for R.
If you're looking for more of a hand-holding approach, I'd recommend\u00a0Gordon Shotwell's free\u00a0Reactive Web Dashboards with Shiny Course\u00a0available on Talk Python Training.
You don't get better at Python by reading about Python or watching videos about Python. There's no lack of good resources for Python exercises. A few resources that I've found particularly useful are listed below.
Corey Schafer's YouTube channel (youtube.com/@coreyms): This channel has playlists of short videos and tutorials on Pandas, Matplotlib, OOP, and many other short and long video walkthroughs I've found useful.
Python on Exercism (exercism.org/tracks/python): Lots of free exercises.
Practice Python (practicepython.org):\u00a0More practice!
Automate the Boring Stuff with Python (automatetheboringstuff.com): A free online book with simple projects for automating boring tasks (renaming files, etc).
Advent of Code (adventofcode.com): An Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. It starts every year around the holidays, but you can go back and view old puzzles from previous years any time. These are fun for little friendly office competition as well.
Of all the advice in this post, this one is likely to get stale fastest, so I'll keep it short. Use LLMs where you can! ChatGPT and other LLMs can essentially act as your pair programmer or personal StackOverflow helping you translate your R expertise into Python with contextualized explanations. You can ask ChatGPT for Python equivalents of familiar R functions, syntax comparisons, or specific code examples tailored to data manipulation, statistical analysis, or visualization. You can paste in Python code, ask what it's doing, and see what equivalent R code would have been. Paste in error messages and your code and it'll tell you what went wrong and how to change things. I imagine all the leading LLMs are better with Python than nearly any other programming language, so make use of these tools where you can!
\u00a0This is reposted from the original at\u00a0https://blog.stephenturner.us/p/use-nanoparquet-instead-of-readr-csv.
Yesterday I wrote about base R vs. dplyr vs. duckdb for a simple summary analysis. In that post I simulated 100 million rows of a dataset and wrote to disk as CSV. I then benchmarked how long it took to read in and compute a simple grouped mean.
One thing I\u00a0didn't\u00a0do here was separate the time it took to read data into memory (for base R and dplyr) versus computing the actual summary. Remember, with DuckDB you don't actually have to read the data into memory for SQL operations you can do with DuckDB or duckplyr.
After I wrote this I downloaded the materials from Hadley Wickham's\u00a0R in Production workshop from posit::conf(2024)\u00a0where he encourages using parquet instead of CSV for saving and sharing data.
Why parquet? From the slides:
Like csv, every language can read and write.
Binary column-oriented format (much faster to read).
Rich type system (including missing values).
Efficient encodings + optional compression (smaller files).
Data stored in \"chunks\" (can work in parallel & don't need to read entire file).
Supports complex data types (e.g. list-columns).
Many modern databases can use parquet files directly.
In these slides I also learned about the\u00a0nanoparquet\u00a0package\u00a0\u2014 a\u00a0zero dependency\u00a0package for reading and writing parquet files in R. Besides all the benefits noted above, parquet is much faster to read and write. And, as opposed to saving as .rds, parquet can easily be passed back and forth between R, Python, and other frameworks.
Let's take a look at how reading and writing parquet files compares with CSV, either with base R or readr.
First, load the libraries I'll use here.
library(tidyverse)\nlibrary(nanoparquet)Next, simulate 100 million rows, 5 columns of numeric, categorical, and date data.
n <- 1e8\nset.seed(42)\ndat <-\n tibble(\n id = sample(n/10, n, replace = TRUE),\n category = sample(letters, n, replace = TRUE),\n value1 = rnorm(n),\n value2 = runif(n, min = 0, max = 1000),\n date = sample(seq.Date(from = as.Date(\"2010-01-01\"),\n to = as.Date(\"2020-12-31\"),\n by = \"day\"),\n size=n,\n replace = TRUE)) |>\n arrange(id, category, date)\nThe data looks like this:
# A tibble: 100,000,000 \u00d7 5\n id category value1 value2 date\n \n 1 1 b -1.26 145. 2020-02-15\n 2 1 g -0.316 157. 2020-06-03\n 3 1 i -1.42 919. 2019-11-24\n 4 1 k 0.592 349. 2011-06-15\n 5 1 l -0.292 650. 2013-03-19\n 6 1 m 0.316 802. 2015-05-25\n 7 1 q 0.112 470. 2020-07-13\n 8 1 t 0.637 236. 2011-02-26\n 9 1 w 1.72 49.5 2016-02-25\n10 2 a -0.572 506. 2018-01-22\n# \u2139 99,999,990 more rows Next, I'll create a list holding the time it takes to read and write using base R, readr, and nanoparquet. Notice that readr will attempt to guess the column types when reading in data, by default using only the first 1,000 rows. This could lead to problems if your first 1,000 rows contained numeric values, but later on in the file you have strings or other data types.
time <- list()\ntime$write.csv <- system.time(write.csv(dat, \"dat.csv\"))\ntime$write_csv <- system.time(write_csv(dat, \"dat.csv\"))\ntime$write_pqt <- system.time(write_parquet(dat, \"dat.parquet\"))\ntime$read.csv <- system.time(read.csv(\"dat.csv\"))\ntime$read_csv <- system.time(read_csv(\"dat.csv\"))\ntime$read_pqt <- system.time(read_parquet(\"dat.parquet\"))Finally, extract the timing into a data frame, then plot.
time |>\n map_dbl(\"elapsed\") |>\n enframe(\"fn\", \"seconds\") |>\n mutate(task=ifelse(grepl(\"write\", fn), \"write\", \"read\")) |>\n mutate(pkg=case_when(\n grepl(\"\\\\.csv$\", fn) ~ \"base_R\",\n grepl(\"_csv$\", fn) ~ \"readr\",\n grepl(\"_pqt$\", fn) ~ \"nanoparquet\"\n )) |>\n mutate(pkg=fct_relevel(pkg, \"base_R\", \"readr\", \"nanoparquet\")) |>\n ggplot(aes(task, seconds, fill=pkg)) +\n geom_col(position=\"dodge\") +\n theme_classic() +\n labs(x=\"Task\", y=\"Time (seconds)\", fill=NULL) + \n theme(legend.position=\"bottom\")The results are compelling:
In tabular form:
task base_R readr nanoparquet\nread 366 21 6\nwrite 231 68 8Writing with nanoparquet is 29x faster than base R, and 8.5x faster than readr. Reading with nanoparquet is 61x faster than base R, 3.5x faster than readr. With the added bonus of column types coming in as the types they were originally created as (id as integer, the date column as an actual date). Additionally, the CSV is 5.5 Gb on disk (which could be compressed), and the parquet file is only 1.9 Gb.
As noted above, another nice feature of using parquet is the ability to store column types as they were originally created, rather than relying on\u00a0readr::read_csv()\u00a0to correctly guess the column type (or supply column types explicitly).
The nanoparquet package has a few other functions that allow us to see various kinds of metadata about the parquet files on disk without actually reading them in. From the\u00a0documentation:
parquet_info()\u00a0shows a basic summary of the file.
parquet_column_types()\u00a0shows the leaf columns, these are are the ones that\u00a0read_parquet()\u00a0reads into R.
parquet_schema()\u00a0shows all columns, including non-leaf columns.
parquet_metadata()\u00a0shows the most complete metadata information: file meta data, the schema, the row groups and column chunks of the file.
> parquet_info(\"dat.parquet\")\n# A data frame: 1 \u00d7 7\n file_name num_cols num_rows ...\n ...\n1 dat.parquet 5 100000000 ...\n\n> parquet_column_types(\"dat.parquet\")\n# A data frame: 5 \u00d7 5\n file_name name type r_type logical_type\n* >\n1 dat.parquet id INT32 integer \n2 dat.parquet category BYTE_ARRAY character \n3 dat.parquet value1 DOUBLE double \n4 dat.parquet value2 DOUBLE double \n5 dat.parquet date INT32 Date Similar to what was demonstrated in the previous post, we can run a query on this data to summarize values grouped by ID for all ~10 million groups in this data using duckplyr as an interface to DuckDB,\u00a0without ever having to actually read the parquet files into memory. Here's how to do that (the\u00a0nrow()\u00a0at the end actually materializes the resulting data frame into a tibble in your R session).
Similar to the\u00a0previous post, this summary analysis took about\u00a03.2 seconds\u00a0on 100 million rows, without actually having to read the data into memory.
library(duckplyr)\nsystem.time({\n res <- \n duckplyr_df_from_parquet(\"dat.parquet\") |>\n summarize(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2),\n .by=id) |>\n arrange(id)\n nrow(res)\n})Results preview:
# A tibble: 9,999,521 \u00d7 3\n id mean_value1 mean_value2\n \n 1 1 0.00983 420.\n 2 2 -0.379 661.\n 3 3 -0.0515 491.\n 4 4 0.0574 542.\n 5 5 -0.173 398.\n 6 6 0.0854 405.\n 7 7 0.200 510.\n 8 8 0.0117 621.\n 9 9 -0.0726 470.\n10 10 -0.456 503.\n# \u2139 9,999,511 more rows If you'd rather write raw SQL instead of using duckplyr, you can do that too. This also takes 3.2 seconds for 100 million rows.
library(DBI)\nlibrary(duckdb)\nsystem.time({\n conn <- dbConnect(duckdb())\n res <-\n dbGetQuery(conn = conn,\n statement=\"\n SELECT id,\n AVG(value1) as mean_value1,\n AVG(value2) as mean_value2\n FROM read_parquet('dat.parquet')\n GROUP BY id\n ORDER BY id\")\n dbDisconnect(conn, shutdown=TRUE)\n})
Reposted from\u00a0https://blog.stephenturner.us/p/duckdb-vs-dplyr-vs-base-r.
TL;DR: For a very simple analysis (means by group on 100M rows), duckdb was 125x faster than base R, and 28x faster than readr+dplyr, without having to read data from disk into memory. The duckplyr package wraps DuckDB's analytical query processing techniques in a dplyr-compatible API. Learn more at\u00a0duckdb.org/docs/api/r\u00a0and\u00a0duckplyr.tidyverse.org.
I wanted to see for myself what the fuss was about with DuckDB.\u00a0DuckDB\u00a0is an open-source, high-performance, in-process SQL query engine designed specifically for analytical tasks. Often referred to as the \"sqlite for analytics\" DuckDB is optimized for OLAP (Online Analytical Processing) workloads \u2014 data is organized by columns meaning that the database is optimized for fast reading and large, complex queries that involve filtering, aggregations, and joins. It's \"embedded\" in the same sense that sqlite is \u2014 it has no external dependencies, no DBMS server to install, setup, and maintain. It doesn't as a separate process \u2014 it runs embedded within a host process, conferring the advantage of high-speed data transfer to and from the database, or in some cases direct querying of the data without actually importing or copying any data at all. Read more at\u00a0duckdb.org/why_duckdb.
DuckDB has an R API available (see\u00a0docs). You can install the duckdb package from\u00a0CRAN\u00a0or\u00a0GitHub. Why might you want to use DuckDB in R?
Large Datasets: When you have data too large for memory, DuckDB can efficiently query data from disk without needing the entire dataset in memory.
SQL-Like Queries: If you're more comfortable with SQL, DuckDB allows you to run SQL queries directly on your R data frames. You can perform complex operations such as grouping, filtering, joining, and aggregating, using familiar SQL syntax. Again, without ever reading data into memory in R.
Efficient Analytical Queries: For workloads involving large aggregations or queries on millions of rows DuckDB can handle large analytical queries while minimizing memory overhead and keeping performance high.
Stay in R!\u00a0Using DuckDB in R lets you use DuckDB for the heavy lifting analytical queries, giving you results back in a table that you can take forward for visualization or further analysis.
I wanted to get a sense for myself how fast DuckDB might be compared to base R and dplyr. This will be a simple experiment: I'll simulate some data and write out to disk, then I'll use base R, dplyr, and duckdb to read in the data, group by a variable, and compute the means of two different columns. Simple. There are other, better benchmarks elsewhere (e.g., see posts\u00a0here\u00a0and\u00a0here\u00a0from Appsilon).
All the code below requires a few libraries loaded.
library(tidyverse)\nlibrary(DBI)\nlibrary(duckdb)The code below simulates 100 million rows and some random values and dates.
n <- 1e8\nset.seed(42)\ndat <-\n tibble(\n id = sample(1:1e7, n, replace = TRUE),\n category = sample(letters, n, replace = TRUE),\n value1 = rnorm(n),\n value2 = runif(n, min = 0, max = 1000),\n date = sample(seq.Date(from = as.Date(\"2010-01-01\"),\n to = as.Date(\"2020-12-31\"),\n by = \"day\"), n, replace = TRUE)) |>\n arrange(id, category, date)\ndat |> write_csv(\"dat.csv\")\nThe first 15 rows looks like this:
id category value1 value2 date \n \n 1 1 b -1.26 145. 2020-02-15\n 2 1 g -0.316 157. 2020-06-03\n 3 1 i -1.42 919. 2019-11-24\n 4 1 k 0.592 349. 2011-06-15\n 5 1 l -0.292 650. 2013-03-19\n 6 1 m 0.316 802. 2015-05-25\n 7 1 q 0.112 470. 2020-07-13\n 8 1 t 0.637 236. 2011-02-26\n 9 1 w 1.72 49.5 2016-02-25\n10 2 a -0.572 506. 2018-01-22\n11 2 c -0.608 440. 2018-10-13\n12 2 c -1.26 553. 2020-09-25\n13 2 g 0.563 773. 2018-02-27\n14 2 l 1.07 933. 2012-07-29\n15 2 l -0.163 363. 2017-10-07 The three functions below takes a file path, reads in the data1, computes the mean of value1 and value2 grouped by id, and returns a list of the summarized result and the time it took to run the analysis.
With base R:
process_base <- function(x) {\n time <- system.time({\n dat <- read.csv(x)\n res <- aggregate(cbind(value1, value2) ~ id, data = dat, FUN = mean)\n res <- res[order(res$id), ]\n })\n return(list(res=res, time=time))\n}With dplyr:
process_dplyr <- function(x) {\n time <- system.time({\n dat <- read_csv(x, show_col_types=FALSE, progress=FALSE)\n res <- dat %>%\n group_by(id) %>%\n summarise(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2)\n )\n })\n return(list(res=res, time=time))\n}With duckdb (see below for a friendlier way to write this with duckplyr):
process_duckdb <- function(x) {\n time <- system.time({\n conn <- dbConnect(duckdb())\n res <-\n dbGetQuery(conn = conn,\n statement=glue::glue(\"\n SELECT id,\n AVG(value1) as mean_value1,\n AVG(value2) as mean_value2\n FROM read_csv('{x}', parallel=true)\n GROUP BY id\n ORDER BY id\"))\n })\n dbDisconnect(conn, shutdown=TRUE)\n return(list(res=res, time=time))\n}The results were striking, shown below. Using duckdb was 125x faster than base R, and 28x faster than readr+dplyr.
duckdb: 3.86 seconds
dplyr: 110 seconds
base: 486 seconds
The DBI +\u00a0dbConnect()\u00a0+\u00a0dbGetQuery()\u00a0with plain SQL above isn't the most ergonomic or idiomatic R way to read and analyze data in R. Good news: the\u00a0duckplyr\u00a0package for R wraps DuckDB's analytical query processing techniques in a dplyr-compatible API. Learn more about it at\u00a0duckplyr.tidyverse.org.
Here I'm using the\u00a0duckplyr_df_from_file()\u00a0function to \"read in\" the data from a file, while the summarize and arrange methods are overwriting the functions from dplyr with the duckplyr version. This operation also takes ~3 seconds, and is much easier to read with the familiar dplyr tools you're used to. Take a look at\u00a0DuckDB's blog post on duckplyr\u00a0to really understand how this works.
library(tidyverse)\nlibrary(duckplyr)\nres <-\n duckplyr_df_from_file(\"dat.csv\", table_function=\"read_csv\") |>\n summarize(\n mean_value1 = mean(value1),\n mean_value2 = mean(value2),\n .by=id) |>\n arrange(id)\nresI recently wrote a short essay on\u00a0Learning in Public.\u00a0Something I'm trying to do more of.
The \"benchmark\" here was a simple and limited one, and I mostly used it as an excuse for me to learn a little about duckdb and how to integrate it into my workflow. In this benchmark I didn't attempt to separate time spent reading data into memory versus time spent processing, and the \"analysis\" was a simple mean by group. I didn't attempt to parallelize or optimize either the base R or dplyr approaches, and I didn't do anything with data.table.
For more extensive benchmarks I'd recommend reading through a few of Appsilon's recent posts on using duckdb in R, for huge data, and with Shiny:
There are also a few talks from last year's posit::conf(2023) on using R with duckdb. Those talks are available on YouTube.
Finally, Hannes M\u00fchleisen (CEO, DuckDB Labs) gave a keynote at this year's posit::conf(2024):\u00a0Data Wrangling [for Python or R] Like a Boss With DuckDB. The talks from this year's conference aren't out yet. The talk's abstract from the\u00a0conference agenda\u00a0is below.
Data wrangling is the thorny hedge that higher powers have placed in front of the enjoyable task of actually analyzing or visualizing data. Common struggles come from importing data from ill-mannered CSV files, the tedious task of orchestrating efficient data transformation, or the inevitable management of changes to tables. Data wrangling is rife with questionable ad-hoc solutions, which can sometimes even make things worse. The design rationale of DuckDB is to support the task of data wrangling by bringing the best of decades of data management research and best practices to the world of interactive data analysis in R or Python. For example, DuckDB has one of the world's most advanced CSV reader, native support for Parquet files and Arrow structures, an efficient parallel vectorized query processing engine, and support for efficient atomic updates to tables. All of this is wrapped up in a zero-dependency package available in a programming language near you for free. In my talk, I will discuss the above as well as the design rationale of DuckDB, which was designed and built in collaboration with the Data Science community in the first place.
Keep an eye out for whenever Posit announces the recordings are publicly available.
In the meantime, the workshop materials from posit::conf(2024) are available, including the \"Databases in R\" workshop, taught by Kirill M\u00fcller.
Workshop materials:\u00a0https://github.com/posit-conf-2024/databases
Slides part 1:\u00a0https://posit-conf-2024.github.io/databases/
Slides part 2:\u00a0https://posit-conf-2024.github.io/databases/part-2
Slides part 3:\u00a0https://posit-conf-2024.github.io/databases/part-3.html
Slides part 4:\u00a0https://posit-conf-2024.github.io/databases/part-4.html
This blog has moved. This is reposted from Paired Ends:
https://blog.stephenturner.us/p/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface
Llama 3.1 405B\u00a0is the first open-source LLM on par with frontier models GPT-4o and Claude 3.5 Sonnet. I've been running the\u00a070B model locally for a while now\u00a0using\u00a0Ollama\u00a0+\u00a0Open WebUI, but you're not going to run the 405B model on your MacBook.
Here I demonstrate how to create and deploy a Llama 3.1 405B-powered chatbot on any GitHub repo in <1 minute on HuggingFace Assistants, using an R package as an example
I'm going to use the\u00a0tfboot\u00a0R package as an example here (paper,\u00a0GitHub). I wrote the tfboot package to provide methods for bootstrapping transcription factor binding site disruption to statistically quantify the impact across gene sets of interest compared to an empirical null distribution. The package is meant to integrate with Bioconductor data structures and workflows on the front end, and Tidyverse-friendly tools on the back end. You can read more about the package in the\u00a0paper.
The 42-second video below demonstrates how to create & deploy your chatbot.
Once you create and activate your model, you'll see an interface that will look familiar if you've ever used ChatGPT or similar LLMs.
From here you can click one of the example prompts, or type your own prompt. Let's give it a try. First, a softball pitch. What does this package do? This should be fairly obvious from the README.
Next, let's get a little basic info on usage. The chatbot looks through the package's RMarkdown vignettes and pulls out a high-level protocol on how to run the analysis. There wasn't much context on what motifbreakR is or what you have to do upstream of running tfboot, but further prompting can help with this.
Finally, let's see what it can tell us about the statistical underpinnings of what the package is doing? Of note, this isn't simply regurgitation of what's in the package documentation or vignettes. It's using a combination of the code and documentation itself and integrating that with general information about bootstrapping, null hypothesis significance testing, and transcription factor binding site disruption analysis.
Keep in mind that any assistant you create will be public. You can play around with the tfboot chatbot\u00a0here. Also, know that the 405B model is extremely resource intensive. A few times the bot would timeout and I'd have to retry the prompt. This happens far less often with the 70B model, and the response times are faster. You might experiment around and see for yourself where the speed/accuracy sweet spot is for your specific needs.
","doi":"https://doi.org/10.59350/5hh2d-tnq35","guid":"tag:blogger.com,1999:blog-6232819486261696035.post-5846799348349524841","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1726012800,"rid":"85ms1-zq218","summary":" This blog has moved. This is reposted from Paired Ends: https://blog.stephenturner.us/p/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface Llama 3.1 405B is the first open-source LLM on par with frontier models GPT-4o and Claude 3.5 Sonnet.","tags":["R"],"title":"Create a free Llama 3.1 405B-powered chatbot on any GitHub repo in 1 minute (cross-posted from Paired Ends)","updated_at":1788597378,"url":"https://gettinggeneticsdone.blogspot.com/2024/09/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface.html","version":"v1"},{"authors":[{"affiliation":[{"name":"Form Bio, Bioinformatics"}],"contributor_roles":[],"family":"Turner","given":"Stephen","url":"https://orcid.org/0000-0001-9140-9028"}],"blog":{"authors":[{"name":"Stephen Turner"}],"community_id":"54a97e5a-ef5c-434a-9637-1cd70f4fed4a","created":1780876800,"current_feed_url":null,"description":"Getting Things Done in Genetics & Bioinformatics Research","doi":"https://doi.org/10.59350/gettinggeneticsdone","favicon":"https://rogue-scholar.org/api/communities/54a97e5a-ef5c-434a-9637-1cd70f4fed4a/logo","feed_format":"application/atom+xml","feed_url":"https://gettinggeneticsdone.blogspot.com/feeds/posts/default","filter":null,"generator":"Blogger","home_page_url":"http://gettinggeneticsdone.blogspot.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"gettinggeneticsdone","status":"inactive","subfield":"1311","title":"Getting Genetics Done","updated":null,"use_api":null},"blog_name":"Getting Genetics Done","blog_slug":"gettinggeneticsdone","content_html":"This blog has moved. This is reposted from Paired Ends:
\n\n
PLANES provides a set of methods for evaluating the plausibility of epidemiological signals and forecasts. The PLANES methods are available in the rplanes R package and Shiny app.
Since early 2023, we've been developing methods to conduct what we are calling\u00a0plausibility analysis of epidemiological signals (PLANES). The motivation for PLANES stems from our work on infectious disease forecasting projects, including\u00a0COVID-19,\u00a0influenza-like illness (ILI), and\u00a0influenza hospitalizations. Near-term forecasts of disease patterns can help guide public health decision-making. We've been fortunate enough to participate in several consortia efforts that openly solicit contributions from modelers using different methods. Those methods are often tailored to use training data that comes from epidemiological reporting systems. The openness of these forecasting \"hubs\" is invaluable, and can lead to\u00a0ensemble forecasts\u00a0that as a whole are greater than the sum of their parts.
But after all we're trying to predict the future, so there are plenty of challenges! To name a few:
These particular challenges underscore a need for review of both the incoming surveillance data and forecasts that are generated. But that review requires human intuition \u2014 a \"gut feeling\" \u2014 and it can be hard to scale.
Enter PLANES. We developed this approach to reduce the burden of human review for epidemiological signals. The aim here is to mirror human intuition, but not replace it. In other words, by flagging\u00a0implausibility\u00a0we are not claiming\u00a0impossibility. Instead, the approach provides a screening mechanism so that humans can take a closer look at the signal data. PLANES is agnostic, such that it can be used across either forecasted or observed signals, for different diseases, and for varying temporal and geographic resolutions. The PLANES methods are described in detail in a manuscript we published in August 2024:
Nagraj VP, Benefield AE, Williams D, & Turner SD. (2024). PLANES: Plausibility Analysis of Epidemiological Signals.\u00a0medRxiv,\u00a010.1101/2024.08.22.24312449.
The concept behind the algorithm is demonstrated in Supplemental Figure A1 from the manuscript (below), which illustrates notional components and impacts.
Once we established the PLANES methods, we moved on to the\u00a0rplanes\u00a0R package to implement this approach.
A human can look at a time series of epidemiological data (e.g., incident influenza hospitalizations) and a probabilistic near-term forecast, and get a good sense of whether the forecast \"looks weird\" or not. PLANES attempts to systematize and formalize this assessment such that it can be automated and scaled. We created multiple components, each of which is a binary (yes/no) assessment of plausibility that maps to some feature in the data. All evaluated components are then combined into an ordinal score. By default, each component is equally weighted in the overall PLANES score. When delivered in the rplanes R package, the user can optionally weight components higher or lower in the scoring scheme.
The difference component checks the magnitude of point-to-point differences for all time steps of the evaluated data. This component can be used on either forecasts or observed signals. If an evaluated signal departs from the prior observation more dramatically than has been seen previously in the time series, then it is flagged as implausible. The function internally computes the maximum observed difference (based on absolute value) to set a threshold, which if exceeded will trigger a flag to be raised by the algorithm. While large and unexpected point-to-point changes may naturally occur in epidemiological signals, this component provides a means to draw attention to the most extreme cases.
The coverage component compares the prediction interval for the first horizon of the evaluated signal to the most recent value in the seed. If the interval does not cover the most recent data point, then the flag is raised as implausible. The width of the interval used for this evaluation can be customized. The narrower the width of the prediction interval, the more sensitive this component will be.
The taper component checks whether the prediction interval for the evaluated signal decreases in width (i.e., certainty increases) as horizons progress. The width of the prediction interval at every horizon is assessed against the previous horizon and if any of the intervals for the earlier horizon is wider a flag is raised. One would expect that there would be more variability in signals forecasted further out in time, and therefore the prediction interval would be wider in later horizons.
The repeat component checks whether consecutive values in an observed or forecasted signal are repeated more than the tolerated number of times (k). This stores the maximum number of consecutive repeats for each location and uses this as the default value for\u00a0k. If the evaluated data exceeds\u00a0k, then the signal is considered implausible and a flag is raised.
The trend component assesses if there is a significant change in the magnitude or direction of the slope for the evaluated signal compared to the most recent data in the seed. Each \"change point\" in the signal is identified using a hierarchical divisive algorithm originally implemented in the\u00a0ecp R package.
While the trend component scans the time series for an inflection point, the shape component assesses the time series for unusual shapes across multiple points. To arrive at the shape assessment, the algorithm first divides the observed seed data into sliding windows to form trajectories. The trajectories are summarized as a set of shapes against which the forecasted trajectory is compared. If the shape of the forecasted trajectory does not match any shapes in the seed data, then the forecast is considered implausible per this component. The core intuition underlying this component is that the shape of future data is more likely to reflect patterns that have previously been observed and less likely to be a novel trajectory. Therefore, it may be useful to flag any novel shapes for review. The PLANES paper describes the two methods we used (dynamic time warping, and differences of consecutive observations) to summarize the shapes of signal trajectories.
The zero component was designed to check if there are any \"sudden\" zeros in the evaluated signal. Whether it is a broken surveillance instrument or miscalibrated forecast, we expect it would be unlikely to observe a zero if it has never been reported in the signal data.
Now that we've covered all of the individual PLANES components, let's take a look at the rplanes R package, which implements functions and a Shiny app to evaluate epidemiological signals and forecasts for these components.
rplanes\u00a0is released under an open-source license, with code, documentation and issue tracking on GitHub:\u00a0https://github.com/signaturescience/rplanes.
The package is also released on CRAN:\u00a0https://cran.r-project.org/package=rplanes.
The package website includes function documentation, reproducible examples, and narrative vignettes describing how to get started as well as advanced usage of the tool:\u00a0https://signaturescience.github.io/rplanes/.
We tried to develop\u00a0rplanes\u00a0to be as intuitive as possible. Before assessing the individual PLANES components that we defined, we needed to ensure that user data was formatted consistently. We also needed a structure to store location-specific baseline characteristics against which we could compare the evaluated signal. With this in mind, we created an object-oriented structure using S3 classes in R. The figure below walks through the workflow for preparing data and analyzing data in\u00a0rplanes.
The rplanes package has several\u00a0vignettes\u00a0illustrating\u00a0basic usage,\u00a0detailed descriptions of the PLANES components,\u00a0interpreting plausibility scores, and the\u00a0rplanes explorer shiny app.
In addition to a programmatic interface, we included a point-and-click version of the tool. This is developed as a Shiny app that is built into\u00a0rplanes. We've translated all of the package functionality to widgets in the app. To launch the app simply run\u00a0rplanes_explorer().
rplanes_explorer(). For details, see the\u00a0rplanes explorer vignette.