Compare commits

..

1 Commits

Author SHA1 Message Date
Amolith 8476d3ae6c
Pass messages from backend to frontend 2024-02-23 19:30:23 -05:00
12 changed files with 161 additions and 213 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 209 KiB

View File

@ -1,3 +0,0 @@
SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
SPDX-License-Identifier: CC0-1.0

2
.gitignore vendored
View File

@ -5,7 +5,7 @@
/willow
/*.csv
/data/
/*.sqlite*
/*.sqlite
/config.toml
/.idea/

140
README.md
View File

@ -12,134 +12,74 @@ SPDX-License-Identifier: CC0-1.0
_Forge-agnostic software release tracker_
![screenshot of willow's current web UI](.files/2024-02-24.png)
![screenshot of willow's current web UI](.files/2023-10-29.png)
_This UI is Amolith's attempt at a balance between simple, pleasant, and
functional. Amolith is not a UX professional and would **very** much welcome
input from someone more knowledgeable!_
_This UI is a minimal proof-of-concept, it's going to change drastically in the
near future._
## What is it?
_If you'd rather watch a short video, Amolith gave a 5-minute [lightning talk on
Willow] at the 2023 Ubuntu Summit._
_If you'd rather watch a video, I gave a [lightning talk on Willow] at the 2023
Ubuntu Summit._
[lightning talk on Willow]: https://youtu.be/XIGxKyekvBQ?t=29900
Willow helps developers, sysadmins, and homelabbers keep up with software
releases across arbitrary forge platforms, including full-featured forges like
GitHub, GitLab, or [Forgejo] as well as more minimal options like [cgit] or
[stagit].
Willow tracks software releases across arbitrary forge platforms by trying to
support one of the very few things they all have in common: the VCS. At the
moment, git is the _only_ supported VCS, but I would be interested in adding
Pijul, Fossil, Mercurial, etc. You can also track releases using RSS feeds.
[Forgejo]: https://forgejo.org/
[cgit]: https://git.zx2c4.com/cgit/
[stagit]: https://codemadness.org/stagit.html
Willow exists because decentralisation can be annoying. One piece of software
can be found on GitHub, another piece on GitLab, one on Bitbucket, a fourth on
SourceHut, and a fifth on the developer's self-hosted Forgejo instance. Forgejo
and GitHub have RSS feeds that only notify you of releases. GitLab doesn't
support RSS feeds for anything, just an API you can poke. Some software updates
might be on the developers' personal blog. Sometimes there are CVEs for specific
software and they get published somewhere completely different before they're
fixed in a release.
It exists because decentralisation, as wonderful as it is, does have some pain
points. One piece of software is on GitHub, another piece is on GitLab, one on
Bitbucket, a fourth on [SourceHut], a fifth on the developer's self-hosted
Forgejo instance.
I want to bring all that scattered information under one roof so a developer or
sysadmin can pop open willow's web UI and immediately see what needs updating
where. I've recorded some of my other ideas and plans in [my wiki].
[SourceHut]: https://sourcehut.org/
The capabilities of each platform can also differ, further complicating the
space. For example, Forgejo and GitHub have APIs and RSS release feeds,
SourceHut has an API and RSS feeds that notify you of _all_ activity in the
repo, GitLab only has an API, and there's no standard for discovering the
capabilities of arbitrary git frontends like [legit].
[legit]: https://github.com/icyphox/legit
And _then_ you have different pieces of information in different places; some
developers might publish release announcements on their personal blog and some
projects might release security advisories on an external platform prior to
publishing a release.
All this important info is scattered all over the internet. Willow brings some
order to that chaos by supporting both RSS and one of the _very_ few things all
the forges and frontends have in common: their **V**ersion **C**ontrol
**S**ystem. At the moment, [Git] is the _only_ supported VCS, but we're
definitely interested in adding support for [Pijul], [Fossil], [Mercurial], and
potentially others.
[Git]: https://git-scm.com/
[Pijul]: https://pijul.org/
[Fossil]: https://www.fossil-scm.org/
[Mercurial]: https://www.mercurial-scm.org/
Amolith (the creator) has recorded some of his other ideas, thoughts, and plans
in [his wiki].
[his wiki]: https://wiki.secluded.site/hypha/willow
[my wiki]: https://wiki.secluded.site/hypha/willow
## Installation and use
**Disclaimers:**
1. Prebuilt binaries will be available with the [v0.0.1] release, greatly
simplifying installation.
2. We consider the project _alpha-quality_. There will be bugs.
3. Amolith has tried to make the web UI accessible, but is unsure of its current
usability.
4. The app is not localised yet and English is the only available language.
5. Help with any/all of the above is most welcome!
_**Note:** prebuilt binaries will be available after I release [v0.0.1]_
[v0.0.1]: https://todo.sr.ht/~amolith/willow?search=status%3Aopen%20label%3A%22v0.0.1%22
[communication platforms]: #contributing
### Installation
This assumes Willow will run on an always-on server, like a VPS.
* Clone the repo with `git clone https://git.sr.ht/~amolith/willow`
* Enter the repo's folder with `cd willow`
* Build the binary with `CGO_ENABLED=0 go build -ldflags="-s -w" -o willow
./cmd`
* Transfer the binary to the server however you like
* Execute the binary with `./willow`
* Edit the config with `vim config.toml`
* Daemonise Willow using systemd or OpenRC or whatever you prefer
* Reverse-proxy the web UI (defaults to `localhost:1313`) with Caddy or NGINX or
whatever you prefer
### Use
* Clone the repo
* Build the binary with `CGO_ENABLED=0 go build -ldflags="-s -w" -o willow ./cmd`
* Upload it to a remote server
* Execute the binary
* Edit the `config.toml`
* Create a user with `./willow -a <username>`
* Open the web UI (defaults to `localhost:1313`, but [installation] had you put
a proxy in front)
* Execute the binary again
* Reverse proxy `http://localhost:1313`
* Open the web UI
* Click `Track new project`
* Fill out the form and press `Next`
* Indicate which version you're currently on and press `Track releases`
* You're now tracking that project's releases!
* Fill out the form
* Indicate which version you're currently on
* That's it!
[installation]: #installation
If you no longer use that project, click the `Delete?` link to remove it, and,
if applicable, Willow's copy of its repo.
If you're no longer running the version Willow says you've selected, click the
`Modify?` link to select a different version.
If there are projects where your selected version does _not_ match what Willow
thinks is latest, they'll show up at the top under the **Outdated projects**
heading and have a link at the bottom of the card to `View release notes`.
Clicking that link populates the right column with those release notes.
If there are projects where your selected version _does_ match what Willow
thinks is latest, they'll show up at the bottom under the **Up-to-date
projects** heading.
Note that I still consider the project to be in _alpha_ state. There will be
bugs; please help fix them!
## Contributing
Contributions are very much welcome! Please take a look at the [ticket
tracker][todo] and see if there's anything you're interested in working on. If
there's specific functionality you'd like to see implemented and it's not
mentioned in the ticket tracker, please describe it through one of the platforms
below so we can discuss its inclusion. If we don't feel like it fits with
mentioned in the ticket tracker, please send a description to the [mailing
list][email] so we can discuss its inclusion. If I don't feel like it fits with
Willow's goals, you're encouraged to fork the project and make whatever changes
you like!
Questions, comments, and patches can always go to the [mailing list][email], but
there's also an [IRC channel][irc] and an [XMPP MUC][xmpp] for real-time
interactions.
Questions, comments, and patches can always be sent to the [mailing
list][email], but I'm also in the [IRC channel][irc]/[XMPP room][xmpp] pretty
much 24/7. I might not see messages right away, so please stick around.
- Email: [~amolith/willow@lists.sr.ht][email]
- IRC: [irc.libera.chat/#willow][irc]

View File

@ -22,9 +22,9 @@ func DeleteProject(db *sql.DB, mu *sync.Mutex, id string) error {
}
// GetProject returns a project from the database
func GetProject(db *sql.DB, id string) (map[string]string, error) {
var name, forge, url, version string
err := db.QueryRow("SELECT name, forge, url, version FROM projects WHERE id = ?", id).Scan(&name, &forge, &url, &version)
func GetProject(db *sql.DB, url string) (map[string]string, error) {
var id, name, forge, version string
err := db.QueryRow("SELECT id, name, forge, version FROM projects WHERE url = ?", url).Scan(&id, &name, &forge, &version)
if err != nil {
return nil, err
}

View File

@ -9,7 +9,7 @@ import (
"sync"
)
// UpsertRelease adds or updates a release for a project with a given ID in the
// UpsertRelease adds or updates a release for a project with a given URL in the
// database
func UpsertRelease(db *sql.DB, mu *sync.Mutex, id, projectID, url, tag, content, date string) error {
mu.Lock()
@ -48,12 +48,11 @@ func GetReleases(db *sql.DB, projectID string) ([]map[string]string, error) {
return nil, err
}
releases = append(releases, map[string]string{
"id": id,
"project_id": projectID,
"url": url,
"tag": tag,
"content": content,
"date": date,
"id": id,
"url": url,
"tag": tag,
"content": content,
"date": date,
})
}
return releases, nil

View File

@ -7,6 +7,7 @@ package git
import (
"errors"
"fmt"
"io"
"net/url"
"os"
"strings"
@ -148,23 +149,42 @@ func RemoveRepo(url string) (err error) {
return err
}
path = path[:strings.LastIndex(path, "/")]
dirs := strings.Split(path, "/")
for range dirs {
// TODO: Check whether the two parent directories are empty and remove them if
// so
for i := 0; i < 2; i++ {
path = strings.TrimSuffix(path, "/")
if path == "data" {
break
}
err = os.Remove(path)
empty, err := dirEmpty(path)
if err != nil {
// This folder likely has data, so might as well save some time by
// not checking the parents we can't delete anyway.
break
return err
}
if empty {
err = os.Remove(path)
if err != nil {
return err
}
}
path = path[:strings.LastIndex(path, "/")]
}
return nil
return err
}
// dirEmpty checks if a directory is empty.
func dirEmpty(name string) (empty bool, err error) {
f, err := os.Open(name)
if err != nil {
return false, err
}
defer f.Close()
_, err = f.Readdirnames(1)
if err == io.EOF {
return true, nil
}
return false, err
}
// stringifyRepo accepts a repository URI string and the corresponding local

View File

@ -7,7 +7,6 @@ package project
import (
"crypto/sha256"
"database/sql"
"errors"
"fmt"
"log"
"sort"
@ -42,33 +41,24 @@ type Release struct {
// GetReleases returns a list of all releases for a project from the database
func GetReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
proj.ID = GenProjectID(proj.URL, proj.Name, proj.Forge)
ret, err := db.GetReleases(dbConn, proj.ID)
if err != nil {
return proj, err
}
// TODO: figure out a clean way to remove this so the home page loads
// immediately.
if len(ret) == 0 {
proj, err = fetchReleases(dbConn, mu, proj)
if err != nil {
return proj, err
}
err = upsertReleases(dbConn, mu, proj.ID, proj.Releases)
if err != nil {
return proj, err
}
return proj, nil
}
for _, row := range ret {
proj.Releases = append(proj.Releases, Release{
ID: row["id"],
ProjectID: proj.ID,
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
ID: row["id"],
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
})
}
proj.Releases = SortReleases(proj.Releases)
@ -93,7 +83,7 @@ func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
URL: release.URL,
Date: release.Date,
})
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
err = upsertRelease(dbConn, mu, p.URL, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -112,7 +102,7 @@ func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
URL: release.URL,
Date: release.Date,
})
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
err = upsertRelease(dbConn, mu, p.URL, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -130,18 +120,12 @@ func SortReleases(releases []Release) []Release {
return releases
}
func SortProjects(projects []Project) []Project {
sort.Slice(projects, func(i, j int) bool {
return strings.ToLower(projects[i].Name) < strings.ToLower(projects[j].Name)
})
return projects
}
// upsertReleases updates or inserts a release in the database
func upsertReleases(dbConn *sql.DB, mu *sync.Mutex, projID string, releases []Release) error {
// upsertRelease updates or inserts a release in the database
func upsertRelease(dbConn *sql.DB, mu *sync.Mutex, url string, releases []Release) error {
for _, release := range releases {
date := release.Date.Format("2006-01-02 15:04:05")
err := db.UpsertRelease(dbConn, mu, release.ID, projID, release.URL, release.Tag, release.Content, date)
id := GenReleaseID(url, release.URL, release.Tag)
err := db.UpsertRelease(dbConn, mu, id, url, release.URL, release.Tag, release.Content, date)
if err != nil {
log.Printf("Error upserting release: %v", err)
return err
@ -172,19 +156,12 @@ func Track(dbConn *sql.DB, mu *sync.Mutex, manualRefresh *chan struct{}, name, u
}
func Untrack(dbConn *sql.DB, mu *sync.Mutex, id string) {
proj, err := db.GetProject(dbConn, id)
if err != nil {
fmt.Println("Error getting project:", err)
}
err = db.DeleteProject(dbConn, mu, proj["id"])
err := db.DeleteProject(dbConn, mu, id)
if err != nil {
fmt.Println("Error deleting project:", err)
}
// TODO: before removing, check whether other tracked projects use the same
// repo
err = git.RemoveRepo(proj["url"])
err = git.RemoveRepo(id)
if err != nil {
log.Println(err)
}
@ -210,7 +187,7 @@ func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, re
return strings.ToLower(projectsList[i].Name) < strings.ToLower(projectsList[j].Name)
})
for i := range projectsList {
err = upsertReleases(dbConn, mu, projectsList[i].ID, projectsList[i].Releases)
err = upsertRelease(dbConn, mu, projectsList[i].URL, projectsList[i].Releases)
if err != nil {
fmt.Println("Error upserting release:", err)
continue
@ -237,26 +214,24 @@ func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, re
}
// GetProject returns a project from the database
func GetProject(dbConn *sql.DB, proj Project) (Project, error) {
projectDB, err := db.GetProject(dbConn, proj.ID)
if err != nil && errors.Is(err, sql.ErrNoRows) {
return proj, nil
} else if err != nil {
return proj, err
func GetProject(dbConn *sql.DB, id string) (Project, error) {
projectDB, err := db.GetProject(dbConn, id)
if err != nil {
return Project{}, err
}
p := Project{
ID: proj.ID,
URL: proj.URL,
Name: proj.Name,
Forge: proj.Forge,
ID: projectDB["id"],
URL: projectDB["url"],
Name: projectDB["name"],
Forge: projectDB["forge"],
Running: projectDB["version"],
}
return p, err
}
// GetProjectWithReleases returns a single project from the database along with its releases
func GetProjectWithReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
project, err := GetProject(dbConn, proj)
func GetProjectWithReleases(dbConn *sql.DB, mu *sync.Mutex, id string) (Project, error) {
project, err := GetProject(dbConn, id)
if err != nil {
return Project{}, err
}
@ -282,7 +257,7 @@ func GetProjects(dbConn *sql.DB) ([]Project, error) {
}
}
return SortProjects(projects), nil
return projects, nil
}
// GetProjectsWithReleases returns a list of all projects and all their releases
@ -301,5 +276,5 @@ func GetProjectsWithReleases(dbConn *sql.DB, mu *sync.Mutex) ([]Project, error)
projects[i].Releases = SortReleases(projects[i].Releases)
}
return SortProjects(projects), nil
return projects, nil
}

View File

@ -21,11 +21,12 @@
<header class="wrapper">
<h1>Willow &nbsp;&nbsp;&nbsp;<span><a href="/logout">Log out</a></span></h1>
<p><a href="/new">Track a new project</a></p>
<p class="message-{{ .MessageType }}">{{ .Message }}</p>
</header>
<div class="two_column">
<div class="projects">
<!-- Range through projects that aren't yet up-to-date -->
{{- range . -}}
{{- range .Projects -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<h2>Outdated projects</h2>
{{- break -}}
@ -34,7 +35,7 @@
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<div id="{{ .ID }}" class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h3>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
<p>Latest: <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a></p>
<p><a href="#{{ (index .Releases 0).ID }}">View release notes</a></p>
@ -43,16 +44,16 @@
{{- end -}}
<!-- Range through projects that _are_ up-to-date -->
{{- range . -}}
{{- range .Projects -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<h2>Up-to-date projects</h2>
{{- break -}}
{{- end -}}
{{- end -}}
{{- range . -}}
{{- range .Projects -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<div class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h3>
<p>You've selected <a href="#{{ (index .Releases 0).ID }}">{{ .Running }}</a>. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
</div>
{{- end -}}
@ -60,7 +61,7 @@
</div>
<div class="release_notes">
<h2>Release notes</h2>
{{- range . -}}
{{- range .Projects -}}
<div id="{{ (index .Releases 0).ID }}" class="release_note card">
<h3>{{ .Name }}: release notes for <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a> <span class="close"><a href="#">&#x2716;</a></span></h3>
{{- if eq .Forge "github" "gitea" "forgejo" -}}

View File

@ -44,7 +44,7 @@
<input type="hidden" name="name" value="{{ .Name }}">
<input type="hidden" name="forge" value="{{ .Forge }}">
<input type="hidden" name="id" value="{{ .ID }}">
<input class="button" type="submit" formaction="/new" value="Track releases">
<input class="button" type="submit" formaction="/new" value="Track future releases">
</form>
<!-- Append these if they ever start limiting RSS entries: `(eq $forge "gitea") (eq $forge "forgejo")` -->
{{- if or (eq $forge "github") -}}

View File

@ -128,7 +128,7 @@ header > h1 > span {
}
.card {
border: 2px solid #424242;
border: 2px solid #ccc;
background: #1c1c1c;
}

View File

@ -29,6 +29,12 @@ type Handler struct {
Mu *sync.Mutex
}
type page struct {
Projects []project.Project
Message string
MessageType string
}
//go:embed static
var fs embed.FS
@ -51,7 +57,12 @@ func (h Handler) RootHandler(w http.ResponseWriter, r *http.Request) {
return
}
tmpl := template.Must(template.ParseFS(fs, "static/home.html"))
if err := tmpl.Execute(w, data); err != nil {
p := page{
Projects: data,
Message: "Hello world",
MessageType: "info",
}
if err := tmpl.Execute(w, p); err != nil {
fmt.Println(err)
}
}
@ -80,34 +91,7 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj := project.Project{
ID: project.GenProjectID(submittedURL, name, forge),
URL: submittedURL,
Name: name,
Forge: forge,
}
proj, err := project.GetProject(h.DbConn, proj)
proj, err := project.GetProject(h.DbConn, submittedURL)
if err != nil && err != sql.ErrNoRows {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte(fmt.Sprintf("Error getting project: %s", err)))
@ -117,6 +101,38 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
if proj.Running == "" {
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj = project.Project{
URL: submittedURL,
Name: name,
Forge: forge,
}
proj.URL = strings.TrimSuffix(proj.URL, ".git")
}
proj.ID = project.GenProjectID(proj.URL, proj.Name, proj.Forge)
proj, err = project.GetReleases(h.DbConn, h.Mu, proj)
if err != nil {
w.WriteHeader(http.StatusBadRequest)