Compare commits

..

No commits in common. "29d659b3331d5208c055e54f14c7164ac04c1141" and "c4cd8acd84ae5476c40d15073cf0ae8e7b326e70" have entirely different histories.

24 changed files with 323 additions and 643 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 209 KiB

View File

@ -1,3 +0,0 @@
SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
SPDX-License-Identifier: CC0-1.0

2
.gitignore vendored
View File

@ -5,7 +5,7 @@
/willow
/*.csv
/data/
/*.sqlite*
/*.sqlite
/config.toml
/.idea/

View File

@ -12,59 +12,41 @@ SPDX-License-Identifier: CC0-1.0
_Forge-agnostic software release tracker_
![screenshot of willow's current web UI](.files/2024-02-24.png)
![screenshot of willow's current web UI](.files/2023-10-29.png)
_This UI is Amolith's attempt at something simple and functional, yet still
friendly and pleasant. Amolith is not a UX professional and would **very** much
welcome input from one!_
_This UI is a minimal proof-of-concept, it's going to change drastically in the
near future._
## What is it?
_If you'd rather watch a short video, Amolith gave a 5-minute [lightning talk on
Willow] at the 2023 Ubuntu Summit._
_If you'd rather watch a video, I gave a [lightning talk on Willow] at the 2023
Ubuntu Summit._
[lightning talk on Willow]: https://youtu.be/XIGxKyekvBQ?t=29900
Willow helps developers, sysadmins, and homelabbers keep up with software
releases across arbitrary forge platforms, including full-featured forges like
GitHub, GitLab, or [Forgejo] as well as more minimal options like [cgit] or
[stagit].
Willow tracks software releases across arbitrary forge platforms by trying to
support one of the very few things they all have in common: the VCS. At the
moment, git is the _only_ supported VCS, but I would be interested in adding
Pijul, Fossil, Mercurial, etc. You can also track releases using RSS feeds.
[Forgejo]: https://forgejo.org/
[cgit]: https://git.zx2c4.com/cgit/
[stagit]: https://codemadness.org/stagit.html
Willow exists because decentralisation can be annoying. One piece of software
can be found on GitHub, another piece on GitLab, one on Bitbucket, a fourth on
SourceHut, and a fifth on the developer's self-hosted Forgejo instance. Forgejo
and GitHub have RSS feeds that only notify you of releases. GitLab doesn't
support RSS feeds for anything, just an API you can poke. Some software updates
might be on the developers' personal blog. Sometimes there are CVEs for specific
software and they get published somewhere completely different before they're
fixed in a release.
It exists because decentralisation, as wonderful as it is, does have some pain
points. One piece of software is on GitHub, another piece is on GitLab, one on
Bitbucket, a fourth on [SourceHut], a fifth on the developer's self-hosted
Forgejo instance. The capabilities of each platform can differ as well, making
the problem even more difficult to solve. Forgejo and GitHub have RSS feeds that
notify you of releases as well as APIs. SourceHut has both an API and firehose
RSS feeds that notify you of _all_ activity in the repo. GitLab only has an API.
Some release announcements might be on the developer's personal blog. Sometimes
there's a CVE announcement prior to a release and those get published on a
different platform entirely. It's a mess to keep up with.
I want to bring all that scattered information under one roof so a developer or
sysadmin can pop open willow's web UI and immediately see what needs updating
where. I've recorded some of my other ideas and plans in [my wiki].
[SourceHut]: https://sourcehut.org/
Willow brings some order to that mess by supporting both RSS and one of the
_very_ few things all the forges have in common: their **V**ersion **C**ontrol
**S**ystem. At the moment, [Git] is the _only_ supported VCS, but we're
definitely interested in adding support for [Pijul], [Fossil], [Mercurial], and
potentially others.
[Git]: https://git-scm.com/
[Pijul]: https://pijul.org/
[Fossil]: https://www.fossil-scm.org/
[Mercurial]: https://www.mercurial-scm.org/
Amolith has recorded some of his other ideas, thoughts, and plans in [his wiki].
[his wiki]: https://wiki.secluded.site/hypha/willow
[my wiki]: https://wiki.secluded.site/hypha/willow
## Installation and use
_**Note:** prebuilt binaries will be available after we release [v0.0.1]_
_**Note:** prebuilt binaries will be available after I release [v0.0.1]_
[v0.0.1]: https://todo.sr.ht/~amolith/willow?search=status%3Aopen%20label%3A%22v0.0.1%22
@ -82,7 +64,7 @@ _**Note:** prebuilt binaries will be available after we release [v0.0.1]_
* Indicate which version you're currently on
* That's it!
Note that we still consider the project to be in _alpha_ state. There _will_ be
Note that I still consider the project to be in _alpha_ state. There will be
bugs; please help fix them!
## Contributing
@ -91,13 +73,13 @@ Contributions are very much welcome! Please take a look at the [ticket
tracker][todo] and see if there's anything you're interested in working on. If
there's specific functionality you'd like to see implemented and it's not
mentioned in the ticket tracker, please send a description to the [mailing
list][email] so we can discuss its inclusion. If we don't feel like it fits with
list][email] so we can discuss its inclusion. If I don't feel like it fits with
Willow's goals, you're encouraged to fork the project and make whatever changes
you like!
Questions, comments, and patches can always go to the [mailing list][email], but
there's also an [IRC channel][irc] and an [XMPP MUC][xmpp] for real-time
interactions.
Questions, comments, and patches can always be sent to the [mailing
list][email], but I'm also in the [IRC channel][irc]/[XMPP room][xmpp] pretty
much 24/7. I might not see messages right away, so please stick around.
- Email: [~amolith/willow@lists.sr.ht][email]
- IRC: [irc.libera.chat/#willow][irc]
@ -118,7 +100,6 @@ section._
``` shell
git config sendemail.to "~amolith/willow@lists.sr.ht"
git config format.subjectPrefix "PATCH willow"
git send-email [HASH]
```

View File

@ -10,7 +10,6 @@ import (
"log"
"net/http"
"os"
"strconv"
"sync"
"git.sr.ht/~amolith/willow/db"
@ -23,8 +22,9 @@ import (
type (
Config struct {
Server server
DBConn string
Server server
CSVLocation string
DBConn string
// TODO: Make cache location configurable
// CacheLocation string
FetchInterval int
@ -90,17 +90,17 @@ func main() {
os.Exit(0)
}
mu := sync.Mutex{}
fmt.Println("Starting refresh loop")
go project.RefreshLoop(dbConn, &mu, config.FetchInterval, &manualRefresh, &req, &res)
go project.RefreshLoop(dbConn, config.FetchInterval, &manualRefresh, &req, &res)
var mutex sync.Mutex
wsHandler := ws.Handler{
DbConn: dbConn,
Mutex: &mutex,
Req: &req,
Res: &res,
ManualRefresh: &manualRefresh,
Mu: &mu,
}
mux := http.NewServeMux()
@ -126,20 +126,6 @@ func main() {
}
func checkConfig() error {
defaultDBConn := "willow.sqlite"
defaultFetchInterval := 3600
defaultListen := "127.0.0.1:1313"
defaultConfig := fmt.Sprintf(`# Path to SQLite database
DBConn = "%s"
# How often to fetch new releases in seconds
## Minimum is %ds to avoid rate limits and unintentional abuse
FetchInterval = %d
[Server]
# Address to listen on
Listen = "%s"`, defaultDBConn, defaultFetchInterval, defaultFetchInterval, defaultListen)
file, err := os.Open(*flagConfig)
if err != nil {
if os.IsNotExist(err) {
@ -149,7 +135,15 @@ Listen = "%s"`, defaultDBConn, defaultFetchInterval, defaultFetchInterval, defau
}
defer file.Close()
_, err = file.WriteString(defaultConfig)
_, err = file.WriteString(`# Path to SQLite database
DBConn = "willow.sqlite"
# How often to fetch new releases in seconds
FetchInterval = 3600
[Server]
# Address to listen on
Listen = "127.0.0.1:1313"
`)
if err != nil {
return err
}
@ -168,19 +162,19 @@ Listen = "%s"`, defaultDBConn, defaultFetchInterval, defaultFetchInterval, defau
return err
}
if config.FetchInterval < defaultFetchInterval {
fmt.Println("Fetch interval is set to", strconv.Itoa(config.FetchInterval), "seconds, but the minimum is", defaultFetchInterval, "seconds, using", strconv.Itoa(defaultFetchInterval)+"s")
config.FetchInterval = defaultFetchInterval
if config.FetchInterval < 10 {
fmt.Println("Fetch interval is set to", config.FetchInterval, "seconds, but the minimum is 10, using 10")
config.FetchInterval = 10
}
if config.Server.Listen == "" {
fmt.Println("No listen address specified, using", defaultListen)
config.Server.Listen = defaultListen
fmt.Println("No listen address specified, using 127.0.0.1:1313")
config.Server.Listen = "127.0.0.1:1313"
}
if config.DBConn == "" {
fmt.Println("No SQLite path specified, using \"" + defaultDBConn + "\"")
config.DBConn = defaultDBConn
fmt.Println("No SQLite path specified, using \"willow.sqlite\"")
config.DBConn = "willow.sqlite"
}
return nil

View File

@ -7,8 +7,6 @@ package db
import (
"database/sql"
_ "embed"
"errors"
"sync"
_ "modernc.org/sqlite"
)
@ -16,25 +14,46 @@ import (
//go:embed sql/schema.sql
var schema string
var mutex = &sync.Mutex{}
// Open opens a connection to the SQLite database
func Open(dbPath string) (*sql.DB, error) {
return sql.Open("sqlite", "file:"+dbPath+"?_pragma=journal_mode%3DWAL")
return sql.Open("sqlite", dbPath)
}
// VerifySchema checks whether the schema has been initalised and initialises it
// if not
func InitialiseDatabase(dbConn *sql.DB) error {
var name string
err := dbConn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name='users'").Scan(&name)
if err != nil && errors.Is(err, sql.ErrNoRows) {
mutex.Lock()
defer mutex.Unlock()
if _, err := dbConn.Exec(schema); err != nil {
return err
}
err := dbConn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name='schema_migrations'").Scan(&name)
if err == nil {
return nil
}
return err
tables := []string{
"users",
"sessions",
"projects",
"releases",
}
for _, table := range tables {
name := ""
err := dbConn.QueryRow(
"SELECT name FROM sqlite_master WHERE type='table' AND name=@table",
sql.Named("table", table),
).Scan(&name)
if err != nil {
if err = loadSchema(dbConn); err != nil {
return err
}
}
}
return nil
}
// loadSchema loads the initial schema into the database
func loadSchema(dbConn *sql.DB) error {
if _, err := dbConn.Exec(schema); err != nil {
return err
}
return nil
}

View File

@ -22,10 +22,6 @@ var (
migration1Up string
//go:embed sql/1_add_project_ids.down.sql
migration1Down string
//go:embed sql/2_swap_project_url_for_id.up.sql
migration2Up string
//go:embed sql/2_swap_project_url_for_id.down.sql
migration2Down string
)
var migrations = [...]migration{
@ -39,13 +35,6 @@ var migrations = [...]migration{
downQuery: migration1Down,
postHook: generateAndInsertProjectIDs,
},
2: {
upQuery: migration2Up,
downQuery: migration2Down,
},
3: {
postHook: correctProjectIDs,
},
}
// Migrate runs all pending migrations

View File

@ -55,35 +55,3 @@ func generateAndInsertProjectIDs(tx *sql.Tx) error {
return nil
}
// Basing the project's ID on when it was created (L37) was a bad idea.
func correctProjectIDs(tx *sql.Tx) error {
rows, err := tx.Query("SELECT id, url, name, forge FROM projects")
if err != nil {
return fmt.Errorf("failed to list projects in projects_tmp: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
old_id string
url string
name string
forge string
)
if err := rows.Scan(&old_id, &url, &name, &forge); err != nil {
return fmt.Errorf("failed to scan row from projects_tmp: %w", err)
}
id := fmt.Sprintf("%x", sha256.Sum256([]byte(url+name+forge)))
_, err = tx.Exec(
"UPDATE projects SET id = @id WHERE id = @old_id",
sql.Named("id", id),
sql.Named("old_id", old_id),
)
if err != nil {
return fmt.Errorf("failed to insert project into projects: %w", err)
}
}
return nil
}

View File

@ -4,32 +4,32 @@
package db
import (
"database/sql"
"sync"
)
import "database/sql"
// CreateProject adds a project to the database
func CreateProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec("INSERT INTO projects (url, name, forge, version) VALUES (?, ?, ?, ?)", url, name, forge, running)
return err
}
// DeleteProject deletes a project from the database
func DeleteProject(db *sql.DB, mu *sync.Mutex, id string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec("DELETE FROM projects WHERE id = ?", id)
func DeleteProject(db *sql.DB, url string) error {
_, err := db.Exec("DELETE FROM projects WHERE url = ?", url)
if err != nil {
return err
}
_, err = db.Exec("DELETE FROM releases WHERE project_id = ?", id)
_, err = db.Exec("DELETE FROM releases WHERE project_url = ?", url)
return err
}
// GetProject returns a project from the database
func GetProject(db *sql.DB, id string) (map[string]string, error) {
var name, forge, url, version string
err := db.QueryRow("SELECT name, forge, url, version FROM projects WHERE id = ?", id).Scan(&name, &forge, &url, &version)
func GetProject(db *sql.DB, url string) (map[string]string, error) {
var name, forge, version string
err := db.QueryRow("SELECT name, forge, version FROM projects WHERE url = ?", url).Scan(&name, &forge, &version)
if err != nil {
return nil, err
}
project := map[string]string{
"id": id,
"name": name,
"url": url,
"forge": forge,
@ -38,23 +38,27 @@ func GetProject(db *sql.DB, id string) (map[string]string, error) {
return project, nil
}
// UpdateProject updates an existing project in the database
func UpdateProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec("UPDATE projects SET name=?, forge=?, version=? WHERE url=?", name, forge, running, url)
return err
}
// UpsertProject adds or updates a project in the database
func UpsertProject(db *sql.DB, mu *sync.Mutex, id, url, name, forge, running string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec(`INSERT INTO projects (id, url, name, forge, version)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(id) DO
func UpsertProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec(`INSERT INTO projects (url, name, forge, version)
VALUES (?, ?, ?, ?)
ON CONFLICT(url) DO
UPDATE SET
name = excluded.name,
forge = excluded.forge,
version = excluded.version;`, id, url, name, forge, running)
version = excluded.version;`, url, name, forge, running)
return err
}
// GetProjects returns a list of all projects in the database
func GetProjects(db *sql.DB) ([]map[string]string, error) {
rows, err := db.Query("SELECT id, name, url, forge, version FROM projects")
rows, err := db.Query("SELECT name, url, forge, version FROM projects")
if err != nil {
return nil, err
}
@ -62,13 +66,12 @@ func GetProjects(db *sql.DB) ([]map[string]string, error) {
var projects []map[string]string
for rows.Next() {
var id, name, url, forge, version string
err = rows.Scan(&id, &name, &url, &forge, &version)
var name, url, forge, version string
err = rows.Scan(&name, &url, &forge, &version)
if err != nil {
return nil, err
}
project := map[string]string{
"id": id,
"name": name,
"url": url,
"forge": forge,

View File

@ -6,29 +6,34 @@ package db
import (
"database/sql"
"sync"
)
// UpsertRelease adds or updates a release for a project with a given ID in the
// AddRelease adds a release for a project with a given URL to the database
// DeleteRelease deletes a release for a project with a given URL from the database
// UpdateRelease updates a release for a project with a given URL in the database
// UpsertRelease adds or updates a release for a project with a given URL in the
// database
func UpsertRelease(db *sql.DB, mu *sync.Mutex, id, projectID, url, tag, content, date string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec(`INSERT INTO releases (id, project_id, url, tag, content, date)
func UpsertRelease(db *sql.DB, id, projectURL, releaseURL, tag, content, date string) error {
_, err := db.Exec(`INSERT INTO releases (id, project_url, release_url, tag, content, date)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO
UPDATE SET
url = excluded.url,
release_url = excluded.release_url,
content = excluded.content,
tag = excluded.tag,
content = excluded.content,
date = excluded.date;`, id, projectID, url, tag, content, date)
date = excluded.date;`, id, projectURL, releaseURL, tag, content, date)
return err
}
// GetReleases returns all releases for a project with a given id from the database
func GetReleases(db *sql.DB, projectID string) ([]map[string]string, error) {
rows, err := db.Query(`SELECT id, url, tag, content, date FROM releases WHERE project_id = ?`, projectID)
// GetRelease returns a release for a project with a given URL from the database
// GetReleases returns all releases for a project with a given URL from the database
func GetReleases(db *sql.DB, projectURL string) ([]map[string]string, error) {
rows, err := db.Query(`SELECT project_url, release_url, tag, content, date FROM releases WHERE project_url = ?`, projectURL)
if err != nil {
return nil, err
}
@ -37,20 +42,19 @@ func GetReleases(db *sql.DB, projectID string) ([]map[string]string, error) {
releases := make([]map[string]string, 0)
for rows.Next() {
var (
id string
url string
tag string
content string
date string
projectURL string
releaseURL string
tag string
content string
date string
)
err := rows.Scan(&id, &url, &tag, &content, &date)
err := rows.Scan(&projectURL, &releaseURL, &tag, &content, &date)
if err != nil {
return nil, err
}
releases = append(releases, map[string]string{
"id": id,
"project_id": projectID,
"url": url,
"projectURL": projectURL,
"releaseURL": releaseURL,
"tag": tag,
"content": content,
"date": date,

View File

@ -1,28 +0,0 @@
-- SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
--
-- SPDX-License-Identifier: CC0-1.0
ALTER TABLE releases RENAME TO releases_tmp;
CREATE TABLE IF NOT EXISTS releases (
id TEXT NOT NULL PRIMARY KEY,
project_url TEXT NOT NULL,
release_url TEXT NOT NULL,
tag TEXT NOT NULL,
content TEXT NOT NULL,
date TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO releases (id, project_url, release_url, tag, content, date)
SELECT
r.id,
p.url,
r.url,
r.tag,
r.content,
r.date
FROM releases_tmp r
JOIN projects p ON r.project_url = p.url;
DROP TABLE releases_tmp;

View File

@ -1,29 +0,0 @@
-- SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
--
-- SPDX-License-Identifier: CC0-1.0
ALTER TABLE releases RENAME TO releases_tmp;
CREATE TABLE IF NOT EXISTS releases
(
id TEXT NOT NULL PRIMARY KEY,
project_id TEXT NOT NULL,
url TEXT NOT NULL,
tag TEXT NOT NULL,
content TEXT NOT NULL,
date TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO releases (id, project_id, url, tag, content, date)
SELECT
r.id,
p.id,
r.release_url,
r.tag,
r.content,
r.date
FROM releases_tmp r
JOIN projects p ON r.project_url = p.url;
DROP TABLE releases_tmp;

View File

@ -12,16 +12,12 @@ import (
// DeleteUser deletes specific user from the database and returns an error if it
// fails
func DeleteUser(db *sql.DB, user string) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("DELETE FROM users WHERE username = ?", user)
return err
}
// CreateUser creates a new user in the database and returns an error if it fails
func CreateUser(db *sql.DB, username, hash, salt string) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("INSERT INTO users (username, hash, salt) VALUES (?, ?, ?)", username, hash, salt)
return err
}
@ -76,8 +72,6 @@ func GetSession(db *sql.DB, session string) (string, time.Time, error) {
// InvalidateSession invalidates a session by setting the expiration date to the
// provided time.
func InvalidateSession(db *sql.DB, session string, expiry time.Time) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("UPDATE sessions SET expires = ? WHERE token = ?", expiry.Format(time.RFC3339), session)
return err
}
@ -85,8 +79,6 @@ func InvalidateSession(db *sql.DB, session string, expiry time.Time) error {
// CreateSession creates a new session in the database and returns an error if
// it fails
func CreateSession(db *sql.DB, username, token string, expiry time.Time) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("INSERT INTO sessions (token, username, expires) VALUES (?, ?, ?)", token, username, expiry.Format(time.RFC3339))
return err
}

View File

@ -7,6 +7,7 @@ package git
import (
"errors"
"fmt"
"io"
"net/url"
"os"
"strings"
@ -148,31 +149,47 @@ func RemoveRepo(url string) (err error) {
return err
}
path = path[:strings.LastIndex(path, "/")]
dirs := strings.Split(path, "/")
for range dirs {
// TODO: Check whether the two parent directories are empty and remove them if
// so
for i := 0; i < 2; i++ {
path = strings.TrimSuffix(path, "/")
if path == "data" {
break
}
err = os.Remove(path)
empty, err := dirEmpty(path)
if err != nil {
// This folder likely has data, so might as well save some time by
// not checking the parents we can't delete anyway.
break
return err
}
if empty {
err = os.Remove(path)
if err != nil {
return err
}
}
path = path[:strings.LastIndex(path, "/")]
}
return nil
return err
}
// dirEmpty checks if a directory is empty.
func dirEmpty(name string) (empty bool, err error) {
f, err := os.Open(name)
if err != nil {
return false, err
}
defer f.Close()
_, err = f.Readdirnames(1)
if err == io.EOF {
return true, nil
}
return false, err
}
// stringifyRepo accepts a repository URI string and the corresponding local
// filesystem path, whether the URI is HTTP, HTTPS, or SSH.
func stringifyRepo(url string) (path string, err error) {
url = strings.TrimSuffix(url, ".git")
url = strings.TrimSuffix(url, "/")
ep, err := transport.NewEndpoint(url)
if err != nil {
return "", err
@ -181,7 +198,7 @@ func stringifyRepo(url string) (path string, err error) {
if ep.Protocol == "http" || ep.Protocol == "https" {
return "data/" + strings.Split(url, "://")[1], nil
} else if ep.Protocol == "ssh" {
return "data/" + ep.Host + "/" + ep.Path, nil
return "data/" + ep.Host + ep.Path, nil
} else {
return "", errors.New("unsupported protocol")
}

View File

@ -1,63 +0,0 @@
// SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
//
// SPDX-License-Identifier: Apache-2.0
package git
import (
"testing"
)
func TestStringifyRepo(t *testing.T) {
wantGitHub := "data/github.com/owner/repo"
wantSourceHut := "data/git.sr.ht/~owner/repo"
tests := []struct {
name string
input string
want string
}{
{
name: "GitHubHTTP",
input: "http://github.com/owner/repo",
want: wantGitHub,
},
{
name: "GitHubHTTPS",
input: "https://github.com/owner/repo",
want: wantGitHub,
},
{
name: "GitHubSSH",
input: "git@github.com:owner/repo",
want: wantGitHub,
},
{
name: "SourceHutHTTP",
input: "http://git.sr.ht/~owner/repo",
want: wantSourceHut,
},
{
name: "SourceHutHTTPS",
input: "https://git.sr.ht/~owner/repo",
want: wantSourceHut,
},
{
name: "SourceHutSSH",
input: "git@git.sr.ht:~owner/repo",
want: wantSourceHut,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, err := stringifyRepo(test.input)
if err != nil {
t.Errorf("stringifyRepo(%s) returned error: %v", test.input, err)
}
if got != test.want {
t.Errorf("stringifyRepo(%s) = %s, want %s", test.input, got, test.want)
}
})
}
}

18
go.mod
View File

@ -10,24 +10,24 @@ toolchain go1.21.3
require (
github.com/BurntSushi/toml v1.3.2
github.com/go-git/go-git/v5 v5.11.0
github.com/go-git/go-git/v5 v5.10.1
github.com/microcosm-cc/bluemonday v1.0.26
github.com/mmcdole/gofeed v1.2.1
github.com/spf13/pflag v1.0.5
golang.org/x/crypto v0.19.0
golang.org/x/term v0.17.0
golang.org/x/crypto v0.16.0
golang.org/x/term v0.15.0
modernc.org/sqlite v1.27.0
)
require (
dario.cat/mergo v1.0.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v1.0.0 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c // indirect
github.com/PuerkitoBio/goquery v1.8.1 // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/cloudflare/circl v1.3.6 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
@ -50,11 +50,11 @@ require (
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/unascribed/FlexVer/go/flexver v1.0.0 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
golang.org/x/mod v0.15.0 // indirect
golang.org/x/net v0.21.0 // indirect
golang.org/x/sys v0.17.0 // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.18.0 // indirect
golang.org/x/tools v0.16.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
lukechampine.com/uint128 v1.3.0 // indirect
modernc.org/cc/v3 v3.41.0 // indirect

21
go.sum
View File

@ -7,8 +7,6 @@ github.com/Microsoft/go-winio v0.6.1 h1:9/kr64B9VUZrLm5YYwbGtUJnMgqWVOdUAXu6Migc
github.com/Microsoft/go-winio v0.6.1/go.mod h1:LRdKpFKfdobln8UmuiYcKPot9D2v6svN5+sAH+4kjUM=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c h1:kMFnB0vCcX7IL/m9Y5LO+KQYv+t1CQOiFe6+SV2J7bE=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/ProtonMail/go-crypto v1.0.0 h1:LRuvITjQWX+WIfr930YHG2HNfjR1uOfyf5vE0kC2U78=
github.com/ProtonMail/go-crypto v1.0.0/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
github.com/acomagu/bufpipe v1.0.4 h1:e3H4WUzM3npvo5uv95QuJM3cQspFNtFBzvJ2oNjKIDQ=
@ -26,8 +24,6 @@ github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7N
github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.6 h1:/xbKIqSHbZXHwkhbrhrt2YOHIwYJlXH94E3tI/gDlUg=
github.com/cloudflare/circl v1.3.6/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU=
github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA=
github.com/cyphar/filepath-securejoin v0.2.4 h1:Ugdm7cg7i6ZK6x3xDF1oEu1nfkyfH53EtKeQYTC3kyg=
github.com/cyphar/filepath-securejoin v0.2.4/go.mod h1:aPGpWjXOXUn2NCNjFvBE6aRxGGx79pTxQpKOJNYHHl4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@ -51,8 +47,6 @@ github.com/go-git/go-git/v5 v5.10.0 h1:F0x3xXrAWmhwtzoCokU4IMPcBdncG+HAAqi9FcOOj
github.com/go-git/go-git/v5 v5.10.0/go.mod h1:1FOZ/pQnqw24ghP2n7cunVl0ON55BsjPYvhWHvZGhoo=
github.com/go-git/go-git/v5 v5.10.1 h1:tu8/D8i+TWxgKpzQ3Vc43e+kkhXqtsZCKI/egajKnxk=
github.com/go-git/go-git/v5 v5.10.1/go.mod h1:uEuHjxkHap8kAl//V5F/nNWwqIYtP/402ddd05mp0wg=
github.com/go-git/go-git/v5 v5.11.0 h1:XIZc1p+8YzypNr34itUfSvYJcv+eYdTnTvOZ2vD3cA4=
github.com/go-git/go-git/v5 v5.11.0/go.mod h1:6GFcX2P3NM7FPBfpePbpLd21XxsgdAt+lKqXmCUiUCY=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
@ -124,7 +118,6 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/unascribed/FlexVer/go/flexver v1.0.0 h1:eaAAWwaT8TiGK75wfEgQRPRVJc1ZIiLTLGUKXxpcs0c=
github.com/unascribed/FlexVer/go/flexver v1.0.0/go.mod h1:OkWZGfmV3DV2ADlgoS7W1+dD1OOci4mEracZCi3ulBk=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
@ -139,16 +132,12 @@ golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc=
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/crypto v0.16.0 h1:mMMrFzRSCF0GvB7Ne27XVtVAaXLrPmgPC7/v0tkwHaY=
golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
golang.org/x/crypto v0.19.0 h1:ENy+Az/9Y1vSrlrvBSyna3PITt4tiZLf7sgCjZBX7Wo=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0=
golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.15.0 h1:SernR4v+D55NyBH2QiEQrlBAnj1ECL6AGrA5+dPaMY8=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
@ -163,8 +152,6 @@ golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c=
golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U=
golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -189,8 +176,6 @@ golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc=
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.17.0 h1:25cE3gD+tdBA7lp7QfhuV+rJiE9YXTcS3VG1SqssI/Y=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
@ -201,8 +186,6 @@ golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek=
golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4=
golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0=
golang.org/x/term v0.17.0 h1:mkTF7LCd6WGJNL3K1Ad7kwxNfYAW6a8a8QqtMblp/4U=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@ -223,10 +206,6 @@ golang.org/x/tools v0.14.0 h1:jvNa2pY0M4r62jkRQ6RwEZZyPcymeL9XZMLBbV7U2nc=
golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg=
golang.org/x/tools v0.16.0 h1:GO788SKMRunPIBCXiQyo2AaexLstOrVhuAL5YwsckQM=
golang.org/x/tools v0.16.0/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0=
golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc=
golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps=
golang.org/x/tools v0.18.0 h1:k8NLag8AGHnn+PHbl7g43CtqZAwG60vZkLqgyZgIHgQ=
golang.org/x/tools v0.18.0/go.mod h1:GL7B4CwcLLeo59yx/9UWWuNOW1n3VZ4f5axWfML7Lcg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

View File

@ -7,12 +7,10 @@ package project
import (
"crypto/sha256"
"database/sql"
"errors"
"fmt"
"log"
"sort"
"strings"
"sync"
"time"
"github.com/unascribed/FlexVer/go/flexver"
@ -23,7 +21,6 @@ import (
)
type Project struct {
ID string
URL string
Name string
Forge string
@ -32,43 +29,30 @@ type Project struct {
}
type Release struct {
ID string
ProjectID string
URL string
Tag string
Content string
Date time.Time
ID string
URL string
Tag string
Content string
Date time.Time
}
// GetReleases returns a list of all releases for a project from the database
func GetReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
proj.ID = GenProjectID(proj.URL, proj.Name, proj.Forge)
ret, err := db.GetReleases(dbConn, proj.ID)
func GetReleases(dbConn *sql.DB, proj Project) (Project, error) {
ret, err := db.GetReleases(dbConn, proj.URL)
if err != nil {
return proj, err
}
if len(ret) == 0 {
proj, err = fetchReleases(dbConn, mu, proj)
if err != nil {
return proj, err
}
err = upsertReleases(dbConn, mu, proj.ID, proj.Releases)
if err != nil {
return proj, err
}
return proj, nil
return fetchReleases(dbConn, proj)
}
for _, row := range ret {
proj.Releases = append(proj.Releases, Release{
ID: row["id"],
ProjectID: proj.ID,
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
})
}
proj.Releases = SortReleases(proj.Releases)
@ -76,7 +60,7 @@ func GetReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error)
}
// fetchReleases fetches releases from a project's forge given its URI
func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
func fetchReleases(dbConn *sql.DB, p Project) (Project, error) {
var err error
switch p.Forge {
case "github", "gitea", "forgejo":
@ -87,13 +71,13 @@ func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
}
for _, release := range rssReleases {
p.Releases = append(p.Releases, Release{
ID: GenReleaseID(p.URL, release.URL, release.Tag),
ID: genReleaseID(p.URL, release.URL, release.Tag),
Tag: release.Tag,
Content: release.Content,
URL: release.URL,
Date: release.Date,
})
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
err = upsert(dbConn, p.URL, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -106,13 +90,13 @@ func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
}
for _, release := range gitReleases {
p.Releases = append(p.Releases, Release{
ID: GenReleaseID(p.URL, release.URL, release.Tag),
ID: genReleaseID(p.URL, release.URL, release.Tag),
Tag: release.Tag,
Content: release.Content,
URL: release.URL,
Date: release.Date,
})
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
err = upsert(dbConn, p.URL, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -130,18 +114,12 @@ func SortReleases(releases []Release) []Release {
return releases
}
func SortProjects(projects []Project) []Project {
sort.Slice(projects, func(i, j int) bool {
return strings.ToLower(projects[i].Name) < strings.ToLower(projects[j].Name)
})
return projects
}
// upsertReleases updates or inserts a release in the database
func upsertReleases(dbConn *sql.DB, mu *sync.Mutex, projID string, releases []Release) error {
// upsert updates or inserts a project release into the database
func upsert(dbConn *sql.DB, url string, releases []Release) error {
for _, release := range releases {
date := release.Date.Format("2006-01-02 15:04:05")
err := db.UpsertRelease(dbConn, mu, release.ID, projID, release.URL, release.Tag, release.Content, date)
id := genReleaseID(url, release.URL, release.Tag)
err := db.UpsertRelease(dbConn, id, url, release.URL, release.Tag, release.Content, date)
if err != nil {
log.Printf("Error upserting release: %v", err)
return err
@ -150,47 +128,34 @@ func upsertReleases(dbConn *sql.DB, mu *sync.Mutex, projID string, releases []Re
return nil
}
// GenReleaseID generates a likely-unique ID from its project's URL, its release's URL, and its tag
func GenReleaseID(projectURL, releaseURL, tag string) string {
func genReleaseID(projectURL, releaseURL, tag string) string {
idByte := sha256.Sum256([]byte(projectURL + releaseURL + tag))
return fmt.Sprintf("%x", idByte)
}
// GenProjectID generates a likely-unique ID from a project's URI, name, and forge
func GenProjectID(url, name, forge string) string {
idByte := sha256.Sum256([]byte(url + name + forge))
return fmt.Sprintf("%x", idByte)
}
func Track(dbConn *sql.DB, mu *sync.Mutex, manualRefresh *chan struct{}, name, url, forge, release string) {
id := GenProjectID(url, name, forge)
err := db.UpsertProject(dbConn, mu, id, url, name, forge, release)
func Track(dbConn *sql.DB, manualRefresh *chan struct{}, name, url, forge, release string) {
err := db.UpsertProject(dbConn, url, name, forge, release)
if err != nil {
fmt.Println("Error upserting project:", err)
}
*manualRefresh <- struct{}{}
}
func Untrack(dbConn *sql.DB, mu *sync.Mutex, id string) {
proj, err := db.GetProject(dbConn, id)
if err != nil {
fmt.Println("Error getting project:", err)
}
err = db.DeleteProject(dbConn, mu, proj["id"])
func Untrack(dbConn *sql.DB, manualRefresh *chan struct{}, url string) {
err := db.DeleteProject(dbConn, url)
if err != nil {
fmt.Println("Error deleting project:", err)
}
// TODO: before removing, check whether other tracked projects use the same
// repo
err = git.RemoveRepo(proj["url"])
*manualRefresh <- struct{}{}
err = git.RemoveRepo(url)
if err != nil {
log.Println(err)
}
}
func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, req *chan struct{}, res *chan []Project) {
func RefreshLoop(dbConn *sql.DB, interval int, manualRefresh, req *chan struct{}, res *chan []Project) {
ticker := time.NewTicker(time.Second * time.Duration(interval))
fetch := func() []Project {
@ -199,7 +164,7 @@ func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, re
fmt.Println("Error getting projects:", err)
}
for i, p := range projectsList {
p, err := fetchReleases(dbConn, mu, p)
p, err := fetchReleases(dbConn, p)
if err != nil {
fmt.Println(err)
continue
@ -210,7 +175,7 @@ func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, re
return strings.ToLower(projectsList[i].Name) < strings.ToLower(projectsList[j].Name)
})
for i := range projectsList {
err = upsertReleases(dbConn, mu, projectsList[i].ID, projectsList[i].Releases)
err = upsert(dbConn, projectsList[i].URL, projectsList[i].Releases)
if err != nil {
fmt.Println("Error upserting release:", err)
continue
@ -237,31 +202,18 @@ func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, re
}
// GetProject returns a project from the database
func GetProject(dbConn *sql.DB, proj Project) (Project, error) {
projectDB, err := db.GetProject(dbConn, proj.ID)
if err != nil && errors.Is(err, sql.ErrNoRows) {
return proj, nil
} else if err != nil {
return proj, err
}
p := Project{
ID: proj.ID,
URL: proj.URL,
Name: proj.Name,
Forge: proj.Forge,
Running: projectDB["version"],
}
return p, err
}
// GetProjectWithReleases returns a single project from the database along with its releases
func GetProjectWithReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
project, err := GetProject(dbConn, proj)
func GetProject(dbConn *sql.DB, url string) (Project, error) {
projectDB, err := db.GetProject(dbConn, url)
if err != nil {
return Project{}, err
}
return GetReleases(dbConn, mu, project)
p := Project{
URL: projectDB["url"],
Name: projectDB["name"],
Forge: projectDB["forge"],
Running: projectDB["version"],
}
return p, err
}
// GetProjects returns a list of all projects from the database
@ -274,7 +226,6 @@ func GetProjects(dbConn *sql.DB) ([]Project, error) {
projects := make([]Project, len(projectsDB))
for i, p := range projectsDB {
projects[i] = Project{
ID: p["id"],
URL: p["url"],
Name: p["name"],
Forge: p["forge"],
@ -282,24 +233,5 @@ func GetProjects(dbConn *sql.DB) ([]Project, error) {
}
}
return SortProjects(projects), nil
}
// GetProjectsWithReleases returns a list of all projects and all their releases
// from the database
func GetProjectsWithReleases(dbConn *sql.DB, mu *sync.Mutex) ([]Project, error) {
projects, err := GetProjects(dbConn)
if err != nil {
return nil, err
}
for i := range projects {
projects[i], err = GetReleases(dbConn, mu, projects[i])
if err != nil {
return nil, err
}
projects[i].Releases = SortReleases(projects[i].Releases)
}
return SortProjects(projects), nil
return projects, nil
}

View File

@ -18,51 +18,19 @@
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body>
<header class="wrapper">
<h1>Willow &nbsp;&nbsp;&nbsp;<span><a href="/logout">Log out</a></span></h1>
<p><a href="/new">Track a new project</a></p>
</header>
<div class="two_column">
<div class="projects">
<!-- Range through projects that aren't yet up-to-date -->
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<h2>Outdated projects</h2>
{{- break -}}
{{- end -}}
{{- end -}}
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<div id="{{ .ID }}" class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
<p>Latest: <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a></p>
<p><a href="#{{ (index .Releases 0).ID }}">View release notes</a></p>
</div>
{{- end -}}
{{- end -}}
<!-- Range through projects that _are_ up-to-date -->
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<h2>Up-to-date projects</h2>
{{- break -}}
{{- end -}}
{{- end -}}
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<div class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<p>You've selected <a href="#{{ (index .Releases 0).ID }}">{{ .Running }}</a>. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
</div>
{{- end -}}
{{- end -}}
</div>
<div class="release_notes">
<h2>Release notes</h2>
{{- range . -}}
<div id="{{ (index .Releases 0).ID }}" class="release_note card">
<h3>{{ .Name }}: release notes for <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a> <span class="close"><a href="#">&#x2716;</a></span></h3>
<h1>Willow &nbsp;&nbsp;&nbsp;<span><a href="/logout">Log out</a></span></h1>
<p><a href="/new">Track a new project</a></p>
<div class="projects">
<!-- Range through projects that aren't yet up-to-date -->
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<div class="project">
<h2><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h2>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
<p>Latest: <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a></p>
<p>
<details>
<summary>Expand release notes</summary>
{{- if eq .Forge "github" "gitea" "forgejo" -}}
{{- (index .Releases 0).Content -}}
{{- else -}}
@ -70,10 +38,20 @@
{{- (index .Releases 0).Content -}}
</pre>
{{- end -}}
<p><a class="return_to_project" href="#{{ .ID }}">Back to project</a></p>
</div>
{{- end -}}
</div>
</details>
</p>
</div>
{{- end -}}
{{- end -}}
<!-- Range through projects that _are_ up-to-date -->
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<div class="project">
<h2><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span style="font-size: 12px;"><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h2>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
</div>
{{- end -}}
{{- end -}}
</body>
</html>

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body class="wrapper">
<body>
<h1>Willow</h1>
<form method="post">
<div class="input">

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body class="wrapper">
<body>
<h1>Willow</h1>
<form method="post">
<div class="input">

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body class="wrapper">
<body>
<h1>Willow</h1>
<form method="post">
<div class="input">
@ -31,9 +31,9 @@
<label for="{{ .Tag }}"><a href="{{ .URL }}">{{ .Tag }}</a></label><br>
{{- else -}}
{{- if eq $forge "sourcehut" -}}
<label for="{{ .Tag }}"><a href="{{ $url }}/refs/{{ .Tag }}">{{ .Tag }}</a></label><br>
<label for="{{ .Tag }}"><a href="{{ $url }}/refs/{{ .Tag }}">{{ .Tag }}</label><br>
{{- else if eq $forge "gitlab" -}}
<label for="{{ .Tag }}"><a href="{{ $url }}/-releases/{{ .Tag }}">{{ .Tag }}</a></label><br>
<label for="{{ .Tag }}"><a href="{{ $url }}/-releases/{{ .Tag }}">{{ .Tag }}</label><br>
{{- else -}}
<label for="{{ .Tag }}">{{ .Tag }}</label><br>
{{- end -}}
@ -43,7 +43,6 @@
<input type="hidden" name="url" value="{{ .URL }}">
<input type="hidden" name="name" value="{{ .Name }}">
<input type="hidden" name="forge" value="{{ .Forge }}">
<input type="hidden" name="id" value="{{ .ID }}">
<input class="button" type="submit" formaction="/new" value="Track future releases">
</form>
<!-- Append these if they ever start limiting RSS entries: `(eq $forge "gitea") (eq $forge "forgejo")` -->

View File

@ -37,11 +37,12 @@
}
html {
max-width: 500px;
margin: auto auto;
color: #2f2f2f;
background: white;
font-family: 'Atkinson Hyperlegible', sans-serif;
scroll-behavior: smooth;
padding-bottom: 20px;
}
a {
@ -52,28 +53,8 @@ a:visited {
color: #0640e0;
}
.two_column {
display: flex;
gap: 30px;
flex-direction: row;
margin: auto auto;
max-width: 1000px;
height: 92vh;
}
.projects, .release_notes {
overflow: scroll;
flex: 0 0 500px;
}
.release_note.card:not(:target) { display: none; }
.release_note.card:target { display: block; }
.return_to_project {
display: none;
}
.card {
.project {
max-width: 500px;
border: 2px solid #2f2f2f;
background: #f8f8f8;
border-radius: 5px;
@ -82,37 +63,36 @@ a:visited {
box-shadow: 0 4px 8px 0 rgba(0,0,0,0.2),0 6px 20px 0 rgba(0,0,0,0.19);
}
.card > h3 {
.project > h2 {
margin-top: 0;
}
.card > p:first-of-type {
.project > p:first-of-type {
margin-bottom: 16px;
}
.card > p:last-of-type {
margin-bottom: 16px;
}
.close, .delete { float: right; }
.delete { font-size: 12px; }
.close > a {
text-decoration: none;
color: #2f2f2f;
}
.card > pre, .card > div > pre { overflow: scroll; }
.wrapper {
max-width: 500px;
margin: auto auto;
}
header > h1 > span {
.project > h2 > span {
font-size: 12px;
float: right;
}
body > h1 > span {
font-size: 12px;
float: right;
}
.project > details > pre {
overflow: scroll;
}
summary {
cursor: pointer;
}
details summary > * {
display: inline;
}
@media (prefers-color-scheme: dark) {
html {
background: #171717;
@ -127,36 +107,8 @@ header > h1 > span {
color: #5582ff;
}
.card {
border: 2px solid #424242;
.project {
border: 2px solid #ccc;
background: #1c1c1c;
}
.close > a {
color: #ccc;
}
}
@media only screen and (max-width: 1000px) {
div[id] {
display: block;
}
.two_column {
flex-direction: column;
}
.projects, .release_notes {
overflow: visible;
flex: 0 0 100%;
}
.return_to_project {
display: block;
}
.close {
display: none;
}
}

104
ws/ws.go
View File

@ -16,17 +16,18 @@ import (
"text/template"
"time"
"git.sr.ht/~amolith/willow/project"
"git.sr.ht/~amolith/willow/users"
"git.sr.ht/~amolith/willow/project"
"github.com/microcosm-cc/bluemonday"
)
type Handler struct {
DbConn *sql.DB
Mutex *sync.Mutex
Req *chan struct{}
ManualRefresh *chan struct{}
Res *chan []project.Project
Mu *sync.Mutex
}
//go:embed static
@ -40,16 +41,8 @@ func (h Handler) RootHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/login", http.StatusSeeOther)
return
}
data, err := project.GetProjectsWithReleases(h.DbConn, h.Mu)
if err != nil {
fmt.Println(err)
w.WriteHeader(http.StatusInternalServerError)
_, err := w.Write([]byte("Internal Server Error"))
if err != nil {
fmt.Println(err)
}
return
}
*h.Req <- struct{}{}
data := <-*h.Res
tmpl := template.Must(template.ParseFS(fs, "static/home.html"))
if err := tmpl.Execute(w, data); err != nil {
fmt.Println(err)
@ -80,34 +73,7 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj := project.Project{
ID: project.GenProjectID(submittedURL, name, forge),
URL: submittedURL,
Name: name,
Forge: forge,
}
proj, err := project.GetProject(h.DbConn, proj)
proj, err := project.GetProject(h.DbConn, submittedURL)
if err != nil && err != sql.ErrNoRows {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte(fmt.Sprintf("Error getting project: %s", err)))
@ -117,7 +83,38 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
proj, err = project.GetReleases(h.DbConn, h.Mu, proj)
if proj.Running == "" {
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj = project.Project{
URL: submittedURL,
Name: name,
Forge: forge,
}
proj.URL = strings.TrimSuffix(proj.URL, ".git")
}
proj, err = project.GetReleases(h.DbConn, proj)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte(fmt.Sprintf("Error getting releases: %s", err)))
@ -132,8 +129,8 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println(err)
}
} else if action == "delete" {
submittedID := params.Get("id")
if submittedID == "" {
submittedURL := params.Get("url")
if submittedURL == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No URL provided"))
if err != nil {
@ -142,7 +139,7 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
project.Untrack(h.DbConn, h.Mu, submittedID)
project.Untrack(h.DbConn, h.ManualRefresh, submittedURL)
http.Redirect(w, r, "/", http.StatusSeeOther)
}
}
@ -152,29 +149,28 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
if err != nil {
fmt.Println(err)
}
idValue := bmStrict.Sanitize(r.FormValue("id"))
nameValue := bmStrict.Sanitize(r.FormValue("name"))
urlValue := bmStrict.Sanitize(r.FormValue("url"))
forgeValue := bmStrict.Sanitize(r.FormValue("forge"))
releaseValue := bmStrict.Sanitize(r.FormValue("release"))
// If releaseValue is not empty, we're updating an existing project
if idValue != "" && nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue != "" {
project.Track(h.DbConn, h.Mu, h.ManualRefresh, nameValue, urlValue, forgeValue, releaseValue)
if nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue != "" {
project.Track(h.DbConn, h.ManualRefresh, nameValue, urlValue, forgeValue, releaseValue)
http.Redirect(w, r, "/", http.StatusSeeOther)
return
}
// If releaseValue is empty, we're creating a new project
if idValue == "" && nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue == "" {
if nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue == "" {
http.Redirect(w, r, "/new?action=yoink&name="+url.QueryEscape(nameValue)+"&url="+url.QueryEscape(urlValue)+"&forge="+url.QueryEscape(forgeValue), http.StatusSeeOther)
return
}
w.WriteHeader(http.StatusBadRequest)
_, err = w.Write([]byte("No data provided"))
if err != nil {
fmt.Println(err)
if nameValue == "" && urlValue == "" && forgeValue == "" && releaseValue == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No data provided"))
if err != nil {
fmt.Println(err)
}
}
}
}