OSDC for Perl developers 2023.01
- Start day: 2023.01.24
- Session 1: Welcome, Version Control, Journal, Slack
- Assignment 1
- Session 2: Create GitHub Pages using the git CLI; GitHub Actions
- Assignment 2
- Session 3: GitHub Actions, CPAN Digger
- Assignment 3
- Session 4: Docker
- Assignment 4
- Session 5: Docher HUB; Docker Compose
- Session 6: GitHub Actions for Types::RENEEB
- Session 7: GitHub Action for DBIx-Class, ack
- Session 8: DBD::Pg, test coverage with Devel::Cover
- Session 9: git stash, detached head
- Session 10: git stash and bisect, adding test to RSRU
- Session 11: Git Flow; GitHub Actions for Test::Class
Session 1: Welcome, Version Control, Journal, Slack
- overview of the course
- Static analysis
- a little about myself
- Self employed
- Introducing testing, CI etc.
- If you'd like to send me an email reply to the one I sent you. Keep the subject line. Remove the irrelevant content. Without this it is very difficult for me to associate all the emails with the different courses I teach.
- Will be in some public GitHub or GitLab repositories
- At the end of each assignment you'll write a report - a blog post / journal entry.
- You will add it to your personal JSON file and send a Pull-Request with the change. (We'll learn these soon)
- First few assignments will be to my projects or your own projects. This allows for quick feedback and integration.
- Then we'll find you open source projects maintained by other people.
- The more you participate, the more you learn in this course.
- The more effort you put in this course, the more you will gain.
- Ask questions!
- Try to help others! The more you help others the more you will learn.
- Grades: (if relevant) are based on the work done during the course. There is no end-project or exam at the end.
- overview of the course
- Why use version control?
- Wikipedia and the version control there. Recommended to watch:
GitHub: process of contributing to an Open Source project using the GitHub web site. Editing and sending a Pull-Request. Use a the
cm-demouser to make a change in the README of this repository and then to add the json file. Show how the CI fails when we add an incorrectly formatted file.
What is JSON?
Show the Git repository of the project and the web site generated from it.
Show blogging platform
- DEV - See my account
We saw the drawing of the GitHub process in the cloud.
We looked at CPAN-Digger
- We looked at the recent on MetaCPAN
- We looked at some of the features of GitHub while looking at the Dancer project. (Insights, list of commits, forks, stars, watch)
- The original Markdown
- GitHub flavored Markdown
- Bullet points
- Video 1-2
- You will have to publish a journal of your process.
- DEV - shared blogging platform. See my account
- Create an account on the blogging platform you selected. (if you already have one, you can use that)
- Create an account on GitHub (if you already have one, use that)
- Create an account on GitLab (if you already have one, use that)
- Add a picture to all these accounts. It is preferably a picture of you, but it can be a drawing of you, or some other avatar you might want to use.
- Send a pull-request to the GitHub repository of the course adding a JSON file. The name of the file should be your GitHub username and it should include key-value pairs as in the example. (The
postswill be an empty list.) Check the result of GitHub Actions.
- Join the Slack workspace (I send invitations to everyone to their email address.) and say hi.
- Write a blog post about the course. In your post link to your GitHub and GitLab accounts and to your Pull-Request. If you encountered any issue, write about that and how you solved it. If you use an avatar instead of your own picture, describe how you created the avatar.
- In the blog post tell us a bit about your background.
- What programming language(s) you use?
- Which interesting 3rd-party libraries do you use? You can mention big ones, but it is probably more interesting if you mention more esoteric ones.
- Include links to the home-page of each project and the GitHub/GitLab repository of each project.
- What would you like to accomplish in the course?
- Which open source projects would you like to contribute to.
Update your Pull-request adding the URL to the blog post to the
postsfield in the json file.
There are 3 GitHub repositories with lists of GitHub organization published by higher education institutions, governments, and corporations. Find at least 5 more organizations that share some of their code using an open source license in GitHub or GitLab. An organization can be a corporation, a university, a college, a research institute, or a government. (e.g. find a list of universities and use the search feature of GitHub to find GitHub organizations that belong to the institute).
A couple of suggestions for the blog posts
* Use a title that can sound interesting to others as well eg. How to contribute to an open source project or How to Send a pull request on GitHub.
osdc tag and other relevant tags.
* Add the
series: field to the
Jekyll front matter (the header of each post on DEV.to)
* Use Markdown in the post.
* Include links to the relevant sites and pages such as the web site of the Open Source Development Course and the web site of our course: Open Source Development Course for Perl developers.
Session 2: Create GitHub Pages using the git CLI; GitHub Actions
HTML - Hyper Text Markup Language
- just view source in a browser
GitHub pages https://cm-demo.github.io/
- Plain Markdown files in the
- Configuring GitHub Actions with Jekyll in the
.github/workflows/folder. Then we changed the source to be
- Plain Markdown files in the
git config --global --add user.name "Foo Bar"
git config --global --add user.email email@example.com
These commands created the
ssh-keygen Add public key to GitHub in User setting area
``` git clone
git status git diff git add git commit git show SHA git push git remote -v
git pull # both with merge and rebase ```
~/.gitconfig set the default action for
rebase = true
We also saw:
- Set up your own website on github pages
Once it is done add the following entry to your JSON file:
mentors/szabgab.json for an example.
- Collect the git repositories of the projects you depend on. If CPAN modules then MetaCPAN might have the link.
- Add them as a list to your JSON file. See the
mentors/szabgab.jsonfor an example.
- Blog about what we learned. Add links. (See my suggestions above how to improve your blog post.) If you feel something is missing from my notes (this file). Feel free to add them with a PR.
Session 3: GitHub Actions, CPAN Digger
- GitHub Actions
- GitHub Actions slides
- GitHub Actions examles
Specifically we looked at * Bash * PostgreSQL * Perl with Makefile.PL * OSDC Site generator
We sent this Pull Request to the Perl-LanguageServer
We looked for the GitHub repository of SPVM-FindBin, checked out another CPAN module of the same author and found the repo here
We had a short intro to clone/branch/push/pr/pull but we'll cover it again.
- Video 3-2
- Find at least 2 Perl modules on CPAN Digger that has "something missing". Send a pull-request to each one of them.
- Look at CPAN Digger
- Some distributions have a link to their GitHub repository but not to the "issues".
- Some distributions don't have a link to their GitHub. In some cases it isn't hard to track down the repository and then you can change it to make it include the links to the GitHub repository and to the issues.
- Blog about them!
Session 4: Docker
We set out working on one of the Perl modules listed in the JSON files in the participants/ folder, but then ended up covering Docker.
We used many examples from the Docker slides to introduce Docker.
We saw the Docker image I use to deal with various open source projects.
We added Dist::Zilla to this image.
- Create a Docker image for yourself.
- Run the tests of one of your favorite modules inside the container.
- Write a blog post linking to the issues you opened and the Pull-Request you sent. Even if they have not been accepted. Then add the link of that blog post to your JSON file in the participants folder.
Session 5: Docker HUB; Docker Compose
- Show how to connect the GitHub repository to Docker hub.
The configuration file of docker to map all the data (all the volumes, images, containers) to a disk which is not the default.
$ cat /etc/docker/daemon.json
- Show one of my real-world projects using Docker compose
- Show editing a project while it is running in Docker compose
Take one of the projects from the list of projects of the participants, run the tests locally using the Docker container (dr). Open issue when necessary. Set up GitHub Actions if needed. Create test coverage report.
We looked at https://github.com/reneeb/Types-RENEEB and found out that Steve already had a fork and that his changes were already applied to the original repo. (The best course of action at this point might be is to remove the fork and create it again.)
- We found that the project had clear instructions on how to set up the development environment and how to run the tests. Nice.
- We also found that some of the tests fail. Not so good. Steve will check why. He will probably open an issue with the failure. Even if later he finds out the reason, it is a good idea to have it documented on GitHub.
- Video 5-2
- Pick one or more projects
- Try to setup the local development environment.
- If you cannot, open an issue asking the author how. (Feel free to mention me by including @szabgab and mentioning the course by including a link to to https://osdc.code-maven.com/
- If you can setup verify that there are clear instructions how to do this in the README file or in some other file (e.g. CONTRIBUTING).
- If there are not, send a PR with the instructions so the next person will have less trouble.
- Run the tests locally.
- If there are failures report them.
- Check if GitHub Actions is configured. If not, then configure it. (or open an issue asking the author if s/he wants it).
- Create test coverage report. Add more tests if possible.
Session 6: GitHub Actions for Types::RENEEB
We sent several pull-requests to Types-RENEEB
- updated files generated by
- fixing a test failure
- Adding Github Actions. We started with the GitHub Action skeleton for Dist::Zilla.
- Renee, the author of this project accepted our Pull-Requests within a few minutes. Then we sent another PR
- enabling more OSes and versions of Perl in GitHub Actions
- updated files generated by
Clone the original repository
git clone firstname.lastname@example.org:reneeb/Types-RENEEB.git
Create a fork of the repository via GitHub UI.
Setup a new git remote to point to the forked repository:
git remote add fork email@example.com:yewtc/Types-RENEEB.git
- Create a branch to prepare a pull-request
git checkout -b BRANCH
git add .
git commit -m "..."
git push --set-upstream fork BRANCH
Send the pull-request from the GitHub UI
Integrate the progress of original repository to our local clone
git checkout master
git pull origin master
- Once the Pull-request was accepted we could delete the branch locally and remotely
git branch -d BRANCH
git push -d origin BRANCH
- We created Test coverage report of the Types-RENEEB pacakge. It was not too interesting as everything was 100%.
``` cpanm Dist::Zilla::App::Command::cover
dzil cover ```
We also sent a pull-request to Devel::Cover with this information.
- Video 6-2
Session 7: GitHub Action for DBIx-Class, ack
In this session we tried to add GitHub Actions (CI) to DBIx::Class.
It already had an extensive configuration of Travis-CI, but unfortunately Travis stopped its free service.
I briefly explained how my slides used to be generated using a Webhook.
We used the playground Docker image we covered in a previous session.
We still had to install Module::Install as it was a developer dependency.
cpanm --notest Module::Install
Then we could install all the other dependencies:
cpanm --installdeps --notest .
However when we ran
perl Makefile.PL it still complained about a lot of missing modules. So we created a temporary file just to install those:
cpanm --notest Class::DBI::Plugin::DeepAbstractSearch
cpanm --notest Class::MethodCache
cpanm --notest Class::Unload
cpanm --notest Date::Simple
cpanm --notest DateTime::Format::MySQL
cpanm --notest DateTime::Format::Pg
cpanm --notest DateTime::Format::SQLite
cpanm --notest DateTime::Format::Strptime
cpanm --notest JSON
cpanm --notest JSON::Any
cpanm --notest JSON::DWIW
cpanm --notest JSON::XS
cpanm --notest Math::Base36
cpanm --notest MooseX::Types::JSON
cpanm --notest MooseX::Types::LoadableClass
cpanm --notest MooseX::Types::Path::Class
cpanm --notest PadWalker
cpanm --notest Pod::Coverage
cpanm --notest SQL::Translator
cpanm --notest Test::EOL
cpanm --notest Test::NoTabs
cpanm --notest Test::Pod
cpanm --notest Test::Pod::Coverage
cpanm --notest Test::Strict
cpanm --notest Text::CSV
cpanm --notest Time::Piece::MySQL
There must be a better way to install all of these, but we could not find the instructions. So one addition to the project could be an easy way to find insttruction on how to set up the local development environment.
After this we managed to run the tests though many were skipped for various reasons. For example some needed access to real databases.
Then we set up CI based on one of the GitHub Actions examples.
We pushed it out, but it did not start to work. Yesterday GitHub had a big outage, so we looked at the GitHub status and indeed, GitHub Actions was yellow. It did not work properly. Eventually it started our job, but it took several minutes to do so.
To our (or at least my) surprise, once GitHub Actions started to run the tests passed on the first attempt.
However we wanted to see how to run the tests with PostgreSQL as well.
For that we had to install DBD::Pg, another module that does not have a CI. (neither to DBD::mysql nor DBD::Oracle for that matter.
To install DBD::Pg locally we found in the readme of DBD::Pg which is linked from the DBD-Pg page. (I did not remember that during our session.)
apt-get install libpq-dev
cpanm --notest DBD::Pg
Then we added another YAML file for the GitHub Action, this time running insied a Docker container, based on the PostgreSQL GitHub Action example.
It needed some parameters to pass the hostname of the Postgres server, the name of the database, the username and the password.
I briefly used and mentioned ack aka. beyondgrep an excellent grep-like tool in Perl.
To our surprise this too worked on the first try. I was so surprised I had to see a more detailed report and thus we split the GitHub Action job up and had a separate section where we used prove to run the tests in verbose mode.
At the end we sent a pull-request with what we accomplished.
Session 8: DBD::Pg, test coverage with Devel::Cover
- Video 8-1
We saw how to setup a development environment for https://metacpan.org/pod/DBD::Pg using Docker and Docker Compose.
- Then we looked at the coverage report of this module using Devel::Cover
We looked at CPANcover
We looked at the CPAN River list of distribution
- We looked at the test coverage report of DateTime::Format::Pg
- We tried to reproduce this issue but got to the conclusion that it looks like an incorrect expectaton from the user.
- Docker compose with PostgreSQL
- We talked about Test-Exception and SKIP.
- I mentioned the Pay It Forward project. Apparently it is an American expression.
- It would be nice to add the test coverage numbers to CPAN-Digger
Session 9: git stash, detached head
git add -i git stash -u git stash list git stash show git stash show -p
Session 10: git stash and bisect, adding test to RSRU
git detached head merge / rebase reset stash bisect revert
In the 2nd video we added a test to RSRU see the Pull-request
- Video 10-2
Session 11: Git Flow; GitHub Actions for Test::Class
We talked about Git Flow vs. having a single main branch. The latter would need a good (acceptance/integration) test coverage so you can trust the test for a release without requiring manual QA. (The manual QA can still work as exploratory testing that also feeds ideas to the people who then convert the test-cases into automated tests.
We looked at the CI system of Test::Class that first creates a release and then uses that release on various versions of Perl to test it. It also tests a number of downstream distributions to verify that the new version does not break them.
- We looked at the GitHub Actions and the source code of CPAN Dashboard
- We looked at the MetaCPAN tools and the experimental dashboard there.
- We looked at the stats page of PyDigger and the mess of the licenses.
- Video 11-2
Session 12: GitHub Actions; faster CI
We talked about frequency of Pull-Requests or code-reviews.
Docker caching - changing the Makefile.PL outside will make the
COPY Makefile.PL .command run again and that will trigger all the steps that follow it.
We talked about CI processes that take too much time to run. Apparently I said earlier that they should run under 10 minutes which of course is just a randome number.
- A few ideas how to deal with CI processes that take too long.
- Ideally we would like to run all the tests on every push and we would like all the tests to finish within 1 second to get feedback quickly.
- Only try to compile Perl scripts if one of the modules change.
- Add more hardware so jobs can be run in parallel.
- Divide the CI to several tiers.
- One that runs fast (1-2 min ?) and hopefully checks crtirical things. Run these on every commit.
- Another one that runs longer (e.g. 10 minutes) and we'll run it only every 10-15 minutes. Maybe we miss some of the pushes.
- Another tier that takes several hours. Run these only once the earlier one finishes.
- Instead of rebuilding the database every time using Perl, dump the data and load it with the tools of the database. That should be much faster.
Cache installations of 3rd party libraries.
- Unfortunately I've forgotten to record the 2nd part.
Session 13: CVEs and lack of maintenance of CPAN modules