Compare commits

...

23 Commits

Author SHA1 Message Date
Amolith 29d659b333
fix link 2024-02-29 16:38:04 -05:00
Amolith b280b19f78
correct links 2024-02-29 16:29:40 -05:00
Amolith 10a28409a4
Improve README wording 2024-02-29 16:28:41 -05:00
Amolith 3e3236101f
Improve README wording 2024-02-29 16:27:06 -05:00
Amolith 01b1e1d37a
Unbold "willow" 2024-02-29 16:25:10 -05:00
Amolith 5fe84d6aff
improve README wording 2024-02-29 16:22:57 -05:00
Amolith 32b5aad675
add missing screenshots 2024-02-29 15:52:49 -05:00
Amolith 9a91d4f2ea
Improve RemoveRepo implementation 2024-02-29 15:51:42 -05:00
Amolith dde4c97802
Improve logic around untracking projects
Frontend was still passing the project's URL to the backend even though
the backend had been refactored to use the project's ID. This is fixed.

Deleting the repo's files should also work now, but I do think I want to
support tracking multiple instances of the same project. This means
we'll need to check for other instances relying on the repo before
deleting it.

We need end-to-end tests 😩😩😩
2024-02-29 15:07:07 -05:00
Amolith bcbc3420a1
Update screenshot 2024-02-24 11:46:18 -05:00
Amolith c3acec928b
Actually reduce card border contrast 2024-02-24 10:20:06 -05:00
Amolith 2398a0384b
Reduce card border contrast 2024-02-24 10:18:00 -05:00
Amolith ecd3635be7
Sort projects alphabetically 2024-02-24 10:08:13 -05:00
Amolith 753b2acf2c
Ignore all SQLite files 2024-02-23 21:24:23 -05:00
Amolith dd7551fd49
Actually finish 🤞 DB-/ID-related refactors 2024-02-23 21:16:50 -05:00
Amolith b6db773ee3
Add ID to project cards so mobile works 2024-02-23 19:24:07 -05:00
Amolith 203cc09590
Add license header to git_test 2024-02-23 18:52:25 -05:00
Amolith 292db50660
Bump git-go and its deps
govulncheck reported that we're calling vulnerable codepaths from an
older version of git-go, so I upgraded git-go and its direct
dependencies.
2024-02-23 18:52:05 -05:00
Amolith d5c7bf70ce
Implement new UI, fix DB use
- Implement dual-column UI
- Swap project table index from URL to ID
- Enable WAL for concurrent reads
- Use a Mutex to protect writes
2024-02-23 18:38:44 -05:00
Amolith 89e894401f
Move hardcoded defaults to top of checkConfig() 2024-02-23 16:46:54 -05:00
Amolith 8cf4a4c284
Add test for stringifyRepo 2024-02-23 16:18:36 -05:00
Amolith edfeefee51
Trim trailing .git, / from repo URIs 2024-02-23 16:00:02 -05:00
Amolith e107cf7338
docs: mention prefix for emailed patches 2024-01-24 15:28:25 -05:00
24 changed files with 641 additions and 321 deletions

BIN
.files/2024-02-24.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 209 KiB

View File

@ -0,0 +1,3 @@
SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
SPDX-License-Identifier: CC0-1.0

2
.gitignore vendored
View File

@ -5,7 +5,7 @@
/willow
/*.csv
/data/
/*.sqlite
/*.sqlite*
/config.toml
/.idea/

View File

@ -12,41 +12,59 @@ SPDX-License-Identifier: CC0-1.0
_Forge-agnostic software release tracker_
![screenshot of willow's current web UI](.files/2023-10-29.png)
![screenshot of willow's current web UI](.files/2024-02-24.png)
_This UI is a minimal proof-of-concept, it's going to change drastically in the
near future._
_This UI is Amolith's attempt at something simple and functional, yet still
friendly and pleasant. Amolith is not a UX professional and would **very** much
welcome input from one!_
## What is it?
_If you'd rather watch a video, I gave a [lightning talk on Willow] at the 2023
Ubuntu Summit._
_If you'd rather watch a short video, Amolith gave a 5-minute [lightning talk on
Willow] at the 2023 Ubuntu Summit._
[lightning talk on Willow]: https://youtu.be/XIGxKyekvBQ?t=29900
Willow tracks software releases across arbitrary forge platforms by trying to
support one of the very few things they all have in common: the VCS. At the
moment, git is the _only_ supported VCS, but I would be interested in adding
Pijul, Fossil, Mercurial, etc. You can also track releases using RSS feeds.
Willow helps developers, sysadmins, and homelabbers keep up with software
releases across arbitrary forge platforms, including full-featured forges like
GitHub, GitLab, or [Forgejo] as well as more minimal options like [cgit] or
[stagit].
Willow exists because decentralisation can be annoying. One piece of software
can be found on GitHub, another piece on GitLab, one on Bitbucket, a fourth on
SourceHut, and a fifth on the developer's self-hosted Forgejo instance. Forgejo
and GitHub have RSS feeds that only notify you of releases. GitLab doesn't
support RSS feeds for anything, just an API you can poke. Some software updates
might be on the developers' personal blog. Sometimes there are CVEs for specific
software and they get published somewhere completely different before they're
fixed in a release.
[Forgejo]: https://forgejo.org/
[cgit]: https://git.zx2c4.com/cgit/
[stagit]: https://codemadness.org/stagit.html
I want to bring all that scattered information under one roof so a developer or
sysadmin can pop open willow's web UI and immediately see what needs updating
where. I've recorded some of my other ideas and plans in [my wiki].
It exists because decentralisation, as wonderful as it is, does have some pain
points. One piece of software is on GitHub, another piece is on GitLab, one on
Bitbucket, a fourth on [SourceHut], a fifth on the developer's self-hosted
Forgejo instance. The capabilities of each platform can differ as well, making
the problem even more difficult to solve. Forgejo and GitHub have RSS feeds that
notify you of releases as well as APIs. SourceHut has both an API and firehose
RSS feeds that notify you of _all_ activity in the repo. GitLab only has an API.
Some release announcements might be on the developer's personal blog. Sometimes
there's a CVE announcement prior to a release and those get published on a
different platform entirely. It's a mess to keep up with.
[my wiki]: https://wiki.secluded.site/hypha/willow
[SourceHut]: https://sourcehut.org/
Willow brings some order to that mess by supporting both RSS and one of the
_very_ few things all the forges have in common: their **V**ersion **C**ontrol
**S**ystem. At the moment, [Git] is the _only_ supported VCS, but we're
definitely interested in adding support for [Pijul], [Fossil], [Mercurial], and
potentially others.
[Git]: https://git-scm.com/
[Pijul]: https://pijul.org/
[Fossil]: https://www.fossil-scm.org/
[Mercurial]: https://www.mercurial-scm.org/
Amolith has recorded some of his other ideas, thoughts, and plans in [his wiki].
[his wiki]: https://wiki.secluded.site/hypha/willow
## Installation and use
_**Note:** prebuilt binaries will be available after I release [v0.0.1]_
_**Note:** prebuilt binaries will be available after we release [v0.0.1]_
[v0.0.1]: https://todo.sr.ht/~amolith/willow?search=status%3Aopen%20label%3A%22v0.0.1%22
@ -64,7 +82,7 @@ _**Note:** prebuilt binaries will be available after I release [v0.0.1]_
* Indicate which version you're currently on
* That's it!
Note that I still consider the project to be in _alpha_ state. There will be
Note that we still consider the project to be in _alpha_ state. There _will_ be
bugs; please help fix them!
## Contributing
@ -73,13 +91,13 @@ Contributions are very much welcome! Please take a look at the [ticket
tracker][todo] and see if there's anything you're interested in working on. If
there's specific functionality you'd like to see implemented and it's not
mentioned in the ticket tracker, please send a description to the [mailing
list][email] so we can discuss its inclusion. If I don't feel like it fits with
list][email] so we can discuss its inclusion. If we don't feel like it fits with
Willow's goals, you're encouraged to fork the project and make whatever changes
you like!
Questions, comments, and patches can always be sent to the [mailing
list][email], but I'm also in the [IRC channel][irc]/[XMPP room][xmpp] pretty
much 24/7. I might not see messages right away, so please stick around.
Questions, comments, and patches can always go to the [mailing list][email], but
there's also an [IRC channel][irc] and an [XMPP MUC][xmpp] for real-time
interactions.
- Email: [~amolith/willow@lists.sr.ht][email]
- IRC: [irc.libera.chat/#willow][irc]
@ -100,6 +118,7 @@ section._
``` shell
git config sendemail.to "~amolith/willow@lists.sr.ht"
git config format.subjectPrefix "PATCH willow"
git send-email [HASH]
```

View File

@ -10,6 +10,7 @@ import (
"log"
"net/http"
"os"
"strconv"
"sync"
"git.sr.ht/~amolith/willow/db"
@ -22,9 +23,8 @@ import (
type (
Config struct {
Server server
CSVLocation string
DBConn string
Server server
DBConn string
// TODO: Make cache location configurable
// CacheLocation string
FetchInterval int
@ -90,17 +90,17 @@ func main() {
os.Exit(0)
}
fmt.Println("Starting refresh loop")
go project.RefreshLoop(dbConn, config.FetchInterval, &manualRefresh, &req, &res)
mu := sync.Mutex{}
var mutex sync.Mutex
fmt.Println("Starting refresh loop")
go project.RefreshLoop(dbConn, &mu, config.FetchInterval, &manualRefresh, &req, &res)
wsHandler := ws.Handler{
DbConn: dbConn,
Mutex: &mutex,
Req: &req,
Res: &res,
ManualRefresh: &manualRefresh,
Mu: &mu,
}
mux := http.NewServeMux()
@ -126,6 +126,20 @@ func main() {
}
func checkConfig() error {
defaultDBConn := "willow.sqlite"
defaultFetchInterval := 3600
defaultListen := "127.0.0.1:1313"
defaultConfig := fmt.Sprintf(`# Path to SQLite database
DBConn = "%s"
# How often to fetch new releases in seconds
## Minimum is %ds to avoid rate limits and unintentional abuse
FetchInterval = %d
[Server]
# Address to listen on
Listen = "%s"`, defaultDBConn, defaultFetchInterval, defaultFetchInterval, defaultListen)
file, err := os.Open(*flagConfig)
if err != nil {
if os.IsNotExist(err) {
@ -135,15 +149,7 @@ func checkConfig() error {
}
defer file.Close()
_, err = file.WriteString(`# Path to SQLite database
DBConn = "willow.sqlite"
# How often to fetch new releases in seconds
FetchInterval = 3600
[Server]
# Address to listen on
Listen = "127.0.0.1:1313"
`)
_, err = file.WriteString(defaultConfig)
if err != nil {
return err
}
@ -162,19 +168,19 @@ Listen = "127.0.0.1:1313"
return err
}
if config.FetchInterval < 10 {
fmt.Println("Fetch interval is set to", config.FetchInterval, "seconds, but the minimum is 10, using 10")
config.FetchInterval = 10
if config.FetchInterval < defaultFetchInterval {
fmt.Println("Fetch interval is set to", strconv.Itoa(config.FetchInterval), "seconds, but the minimum is", defaultFetchInterval, "seconds, using", strconv.Itoa(defaultFetchInterval)+"s")
config.FetchInterval = defaultFetchInterval
}
if config.Server.Listen == "" {
fmt.Println("No listen address specified, using 127.0.0.1:1313")
config.Server.Listen = "127.0.0.1:1313"
fmt.Println("No listen address specified, using", defaultListen)
config.Server.Listen = defaultListen
}
if config.DBConn == "" {
fmt.Println("No SQLite path specified, using \"willow.sqlite\"")
config.DBConn = "willow.sqlite"
fmt.Println("No SQLite path specified, using \"" + defaultDBConn + "\"")
config.DBConn = defaultDBConn
}
return nil

View File

@ -7,6 +7,8 @@ package db
import (
"database/sql"
_ "embed"
"errors"
"sync"
_ "modernc.org/sqlite"
)
@ -14,46 +16,25 @@ import (
//go:embed sql/schema.sql
var schema string
var mutex = &sync.Mutex{}
// Open opens a connection to the SQLite database
func Open(dbPath string) (*sql.DB, error) {
return sql.Open("sqlite", dbPath)
return sql.Open("sqlite", "file:"+dbPath+"?_pragma=journal_mode%3DWAL")
}
// VerifySchema checks whether the schema has been initalised and initialises it
// if not
func InitialiseDatabase(dbConn *sql.DB) error {
var name string
err := dbConn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name='schema_migrations'").Scan(&name)
if err == nil {
err := dbConn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name='users'").Scan(&name)
if err != nil && errors.Is(err, sql.ErrNoRows) {
mutex.Lock()
defer mutex.Unlock()
if _, err := dbConn.Exec(schema); err != nil {
return err
}
return nil
}
tables := []string{
"users",
"sessions",
"projects",
"releases",
}
for _, table := range tables {
name := ""
err := dbConn.QueryRow(
"SELECT name FROM sqlite_master WHERE type='table' AND name=@table",
sql.Named("table", table),
).Scan(&name)
if err != nil {
if err = loadSchema(dbConn); err != nil {
return err
}
}
}
return nil
}
// loadSchema loads the initial schema into the database
func loadSchema(dbConn *sql.DB) error {
if _, err := dbConn.Exec(schema); err != nil {
return err
}
return nil
return err
}

View File

@ -22,6 +22,10 @@ var (
migration1Up string
//go:embed sql/1_add_project_ids.down.sql
migration1Down string
//go:embed sql/2_swap_project_url_for_id.up.sql
migration2Up string
//go:embed sql/2_swap_project_url_for_id.down.sql
migration2Down string
)
var migrations = [...]migration{
@ -35,6 +39,13 @@ var migrations = [...]migration{
downQuery: migration1Down,
postHook: generateAndInsertProjectIDs,
},
2: {
upQuery: migration2Up,
downQuery: migration2Down,
},
3: {
postHook: correctProjectIDs,
},
}
// Migrate runs all pending migrations

View File

@ -55,3 +55,35 @@ func generateAndInsertProjectIDs(tx *sql.Tx) error {
return nil
}
// Basing the project's ID on when it was created (L37) was a bad idea.
func correctProjectIDs(tx *sql.Tx) error {
rows, err := tx.Query("SELECT id, url, name, forge FROM projects")
if err != nil {
return fmt.Errorf("failed to list projects in projects_tmp: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
old_id string
url string
name string
forge string
)
if err := rows.Scan(&old_id, &url, &name, &forge); err != nil {
return fmt.Errorf("failed to scan row from projects_tmp: %w", err)
}
id := fmt.Sprintf("%x", sha256.Sum256([]byte(url+name+forge)))
_, err = tx.Exec(
"UPDATE projects SET id = @id WHERE id = @old_id",
sql.Named("id", id),
sql.Named("old_id", old_id),
)
if err != nil {
return fmt.Errorf("failed to insert project into projects: %w", err)
}
}
return nil
}

View File

@ -4,32 +4,32 @@
package db
import "database/sql"
// CreateProject adds a project to the database
func CreateProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec("INSERT INTO projects (url, name, forge, version) VALUES (?, ?, ?, ?)", url, name, forge, running)
return err
}
import (
"database/sql"
"sync"
)
// DeleteProject deletes a project from the database
func DeleteProject(db *sql.DB, url string) error {
_, err := db.Exec("DELETE FROM projects WHERE url = ?", url)
func DeleteProject(db *sql.DB, mu *sync.Mutex, id string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec("DELETE FROM projects WHERE id = ?", id)
if err != nil {
return err
}
_, err = db.Exec("DELETE FROM releases WHERE project_url = ?", url)
_, err = db.Exec("DELETE FROM releases WHERE project_id = ?", id)
return err
}
// GetProject returns a project from the database
func GetProject(db *sql.DB, url string) (map[string]string, error) {
var name, forge, version string
err := db.QueryRow("SELECT name, forge, version FROM projects WHERE url = ?", url).Scan(&name, &forge, &version)
func GetProject(db *sql.DB, id string) (map[string]string, error) {
var name, forge, url, version string
err := db.QueryRow("SELECT name, forge, url, version FROM projects WHERE id = ?", id).Scan(&name, &forge, &url, &version)
if err != nil {
return nil, err
}
project := map[string]string{
"id": id,
"name": name,
"url": url,
"forge": forge,
@ -38,27 +38,23 @@ func GetProject(db *sql.DB, url string) (map[string]string, error) {
return project, nil
}
// UpdateProject updates an existing project in the database
func UpdateProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec("UPDATE projects SET name=?, forge=?, version=? WHERE url=?", name, forge, running, url)
return err
}
// UpsertProject adds or updates a project in the database
func UpsertProject(db *sql.DB, url, name, forge, running string) error {
_, err := db.Exec(`INSERT INTO projects (url, name, forge, version)
VALUES (?, ?, ?, ?)
ON CONFLICT(url) DO
func UpsertProject(db *sql.DB, mu *sync.Mutex, id, url, name, forge, running string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec(`INSERT INTO projects (id, url, name, forge, version)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(id) DO
UPDATE SET
name = excluded.name,
forge = excluded.forge,
version = excluded.version;`, url, name, forge, running)
version = excluded.version;`, id, url, name, forge, running)
return err
}
// GetProjects returns a list of all projects in the database
func GetProjects(db *sql.DB) ([]map[string]string, error) {
rows, err := db.Query("SELECT name, url, forge, version FROM projects")
rows, err := db.Query("SELECT id, name, url, forge, version FROM projects")
if err != nil {
return nil, err
}
@ -66,12 +62,13 @@ func GetProjects(db *sql.DB) ([]map[string]string, error) {
var projects []map[string]string
for rows.Next() {
var name, url, forge, version string
err = rows.Scan(&name, &url, &forge, &version)
var id, name, url, forge, version string
err = rows.Scan(&id, &name, &url, &forge, &version)
if err != nil {
return nil, err
}
project := map[string]string{
"id": id,
"name": name,
"url": url,
"forge": forge,

View File

@ -6,34 +6,29 @@ package db
import (
"database/sql"
"sync"
)
// AddRelease adds a release for a project with a given URL to the database
// DeleteRelease deletes a release for a project with a given URL from the database
// UpdateRelease updates a release for a project with a given URL in the database
// UpsertRelease adds or updates a release for a project with a given URL in the
// UpsertRelease adds or updates a release for a project with a given ID in the
// database
func UpsertRelease(db *sql.DB, id, projectURL, releaseURL, tag, content, date string) error {
_, err := db.Exec(`INSERT INTO releases (id, project_url, release_url, tag, content, date)
func UpsertRelease(db *sql.DB, mu *sync.Mutex, id, projectID, url, tag, content, date string) error {
mu.Lock()
defer mu.Unlock()
_, err := db.Exec(`INSERT INTO releases (id, project_id, url, tag, content, date)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO
UPDATE SET
release_url = excluded.release_url,
url = excluded.url,
content = excluded.content,
tag = excluded.tag,
content = excluded.content,
date = excluded.date;`, id, projectURL, releaseURL, tag, content, date)
date = excluded.date;`, id, projectID, url, tag, content, date)
return err
}
// GetRelease returns a release for a project with a given URL from the database
// GetReleases returns all releases for a project with a given URL from the database
func GetReleases(db *sql.DB, projectURL string) ([]map[string]string, error) {
rows, err := db.Query(`SELECT project_url, release_url, tag, content, date FROM releases WHERE project_url = ?`, projectURL)
// GetReleases returns all releases for a project with a given id from the database
func GetReleases(db *sql.DB, projectID string) ([]map[string]string, error) {
rows, err := db.Query(`SELECT id, url, tag, content, date FROM releases WHERE project_id = ?`, projectID)
if err != nil {
return nil, err
}
@ -42,19 +37,20 @@ func GetReleases(db *sql.DB, projectURL string) ([]map[string]string, error) {
releases := make([]map[string]string, 0)
for rows.Next() {
var (
projectURL string
releaseURL string
tag string
content string
date string
id string
url string
tag string
content string
date string
)
err := rows.Scan(&projectURL, &releaseURL, &tag, &content, &date)
err := rows.Scan(&id, &url, &tag, &content, &date)
if err != nil {
return nil, err
}
releases = append(releases, map[string]string{
"projectURL": projectURL,
"releaseURL": releaseURL,
"id": id,
"project_id": projectID,
"url": url,
"tag": tag,
"content": content,
"date": date,

View File

@ -0,0 +1,28 @@
-- SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
--
-- SPDX-License-Identifier: CC0-1.0
ALTER TABLE releases RENAME TO releases_tmp;
CREATE TABLE IF NOT EXISTS releases (
id TEXT NOT NULL PRIMARY KEY,
project_url TEXT NOT NULL,
release_url TEXT NOT NULL,
tag TEXT NOT NULL,
content TEXT NOT NULL,
date TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO releases (id, project_url, release_url, tag, content, date)
SELECT
r.id,
p.url,
r.url,
r.tag,
r.content,
r.date
FROM releases_tmp r
JOIN projects p ON r.project_url = p.url;
DROP TABLE releases_tmp;

View File

@ -0,0 +1,29 @@
-- SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
--
-- SPDX-License-Identifier: CC0-1.0
ALTER TABLE releases RENAME TO releases_tmp;
CREATE TABLE IF NOT EXISTS releases
(
id TEXT NOT NULL PRIMARY KEY,
project_id TEXT NOT NULL,
url TEXT NOT NULL,
tag TEXT NOT NULL,
content TEXT NOT NULL,
date TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO releases (id, project_id, url, tag, content, date)
SELECT
r.id,
p.id,
r.release_url,
r.tag,
r.content,
r.date
FROM releases_tmp r
JOIN projects p ON r.project_url = p.url;
DROP TABLE releases_tmp;

View File

@ -12,12 +12,16 @@ import (
// DeleteUser deletes specific user from the database and returns an error if it
// fails
func DeleteUser(db *sql.DB, user string) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("DELETE FROM users WHERE username = ?", user)
return err
}
// CreateUser creates a new user in the database and returns an error if it fails
func CreateUser(db *sql.DB, username, hash, salt string) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("INSERT INTO users (username, hash, salt) VALUES (?, ?, ?)", username, hash, salt)
return err
}
@ -72,6 +76,8 @@ func GetSession(db *sql.DB, session string) (string, time.Time, error) {
// InvalidateSession invalidates a session by setting the expiration date to the
// provided time.
func InvalidateSession(db *sql.DB, session string, expiry time.Time) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("UPDATE sessions SET expires = ? WHERE token = ?", expiry.Format(time.RFC3339), session)
return err
}
@ -79,6 +85,8 @@ func InvalidateSession(db *sql.DB, session string, expiry time.Time) error {
// CreateSession creates a new session in the database and returns an error if
// it fails
func CreateSession(db *sql.DB, username, token string, expiry time.Time) error {
mutex.Lock()
defer mutex.Unlock()
_, err := db.Exec("INSERT INTO sessions (token, username, expires) VALUES (?, ?, ?)", token, username, expiry.Format(time.RFC3339))
return err
}

View File

@ -7,7 +7,6 @@ package git
import (
"errors"
"fmt"
"io"
"net/url"
"os"
"strings"
@ -149,47 +148,31 @@ func RemoveRepo(url string) (err error) {
return err
}
// TODO: Check whether the two parent directories are empty and remove them if
// so
for i := 0; i < 2; i++ {
path = strings.TrimSuffix(path, "/")
path = path[:strings.LastIndex(path, "/")]
dirs := strings.Split(path, "/")
for range dirs {
if path == "data" {
break
}
empty, err := dirEmpty(path)
err = os.Remove(path)
if err != nil {
return err
}
if empty {
err = os.Remove(path)
if err != nil {
return err
}
// This folder likely has data, so might as well save some time by
// not checking the parents we can't delete anyway.
break
}
path = path[:strings.LastIndex(path, "/")]
}
return err
}
// dirEmpty checks if a directory is empty.
func dirEmpty(name string) (empty bool, err error) {
f, err := os.Open(name)
if err != nil {
return false, err
}
defer f.Close()
_, err = f.Readdirnames(1)
if err == io.EOF {
return true, nil
}
return false, err
return nil
}
// stringifyRepo accepts a repository URI string and the corresponding local
// filesystem path, whether the URI is HTTP, HTTPS, or SSH.
func stringifyRepo(url string) (path string, err error) {
url = strings.TrimSuffix(url, ".git")
url = strings.TrimSuffix(url, "/")
ep, err := transport.NewEndpoint(url)
if err != nil {
return "", err
@ -198,7 +181,7 @@ func stringifyRepo(url string) (path string, err error) {
if ep.Protocol == "http" || ep.Protocol == "https" {
return "data/" + strings.Split(url, "://")[1], nil
} else if ep.Protocol == "ssh" {
return "data/" + ep.Host + ep.Path, nil
return "data/" + ep.Host + "/" + ep.Path, nil
} else {
return "", errors.New("unsupported protocol")
}

63
git/git_test.go Normal file
View File

@ -0,0 +1,63 @@
// SPDX-FileCopyrightText: Amolith <amolith@secluded.site>
//
// SPDX-License-Identifier: Apache-2.0
package git
import (
"testing"
)
func TestStringifyRepo(t *testing.T) {
wantGitHub := "data/github.com/owner/repo"
wantSourceHut := "data/git.sr.ht/~owner/repo"
tests := []struct {
name string
input string
want string
}{
{
name: "GitHubHTTP",
input: "http://github.com/owner/repo",
want: wantGitHub,
},
{
name: "GitHubHTTPS",
input: "https://github.com/owner/repo",
want: wantGitHub,
},
{
name: "GitHubSSH",
input: "git@github.com:owner/repo",
want: wantGitHub,
},
{
name: "SourceHutHTTP",
input: "http://git.sr.ht/~owner/repo",
want: wantSourceHut,
},
{
name: "SourceHutHTTPS",
input: "https://git.sr.ht/~owner/repo",
want: wantSourceHut,
},
{
name: "SourceHutSSH",
input: "git@git.sr.ht:~owner/repo",
want: wantSourceHut,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, err := stringifyRepo(test.input)
if err != nil {
t.Errorf("stringifyRepo(%s) returned error: %v", test.input, err)
}
if got != test.want {
t.Errorf("stringifyRepo(%s) = %s, want %s", test.input, got, test.want)
}
})
}
}

18
go.mod
View File

@ -10,24 +10,24 @@ toolchain go1.21.3
require (
github.com/BurntSushi/toml v1.3.2
github.com/go-git/go-git/v5 v5.10.1
github.com/go-git/go-git/v5 v5.11.0
github.com/microcosm-cc/bluemonday v1.0.26
github.com/mmcdole/gofeed v1.2.1
github.com/spf13/pflag v1.0.5
golang.org/x/crypto v0.16.0
golang.org/x/term v0.15.0
golang.org/x/crypto v0.19.0
golang.org/x/term v0.17.0
modernc.org/sqlite v1.27.0
)
require (
dario.cat/mergo v1.0.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c // indirect
github.com/ProtonMail/go-crypto v1.0.0 // indirect
github.com/PuerkitoBio/goquery v1.8.1 // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/cloudflare/circl v1.3.6 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
@ -50,11 +50,11 @@ require (
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/unascribed/FlexVer/go/flexver v1.0.0 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/mod v0.15.0 // indirect
golang.org/x/net v0.21.0 // indirect
golang.org/x/sys v0.17.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.16.0 // indirect
golang.org/x/tools v0.18.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
lukechampine.com/uint128 v1.3.0 // indirect
modernc.org/cc/v3 v3.41.0 // indirect

21
go.sum
View File

@ -7,6 +7,8 @@ github.com/Microsoft/go-winio v0.6.1 h1:9/kr64B9VUZrLm5YYwbGtUJnMgqWVOdUAXu6Migc
github.com/Microsoft/go-winio v0.6.1/go.mod h1:LRdKpFKfdobln8UmuiYcKPot9D2v6svN5+sAH+4kjUM=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c h1:kMFnB0vCcX7IL/m9Y5LO+KQYv+t1CQOiFe6+SV2J7bE=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/ProtonMail/go-crypto v1.0.0 h1:LRuvITjQWX+WIfr930YHG2HNfjR1uOfyf5vE0kC2U78=
github.com/ProtonMail/go-crypto v1.0.0/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
github.com/acomagu/bufpipe v1.0.4 h1:e3H4WUzM3npvo5uv95QuJM3cQspFNtFBzvJ2oNjKIDQ=
@ -24,6 +26,8 @@ github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7N
github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.6 h1:/xbKIqSHbZXHwkhbrhrt2YOHIwYJlXH94E3tI/gDlUg=
github.com/cloudflare/circl v1.3.6/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU=
github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA=
github.com/cyphar/filepath-securejoin v0.2.4 h1:Ugdm7cg7i6ZK6x3xDF1oEu1nfkyfH53EtKeQYTC3kyg=
github.com/cyphar/filepath-securejoin v0.2.4/go.mod h1:aPGpWjXOXUn2NCNjFvBE6aRxGGx79pTxQpKOJNYHHl4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@ -47,6 +51,8 @@ github.com/go-git/go-git/v5 v5.10.0 h1:F0x3xXrAWmhwtzoCokU4IMPcBdncG+HAAqi9FcOOj
github.com/go-git/go-git/v5 v5.10.0/go.mod h1:1FOZ/pQnqw24ghP2n7cunVl0ON55BsjPYvhWHvZGhoo=
github.com/go-git/go-git/v5 v5.10.1 h1:tu8/D8i+TWxgKpzQ3Vc43e+kkhXqtsZCKI/egajKnxk=
github.com/go-git/go-git/v5 v5.10.1/go.mod h1:uEuHjxkHap8kAl//V5F/nNWwqIYtP/402ddd05mp0wg=
github.com/go-git/go-git/v5 v5.11.0 h1:XIZc1p+8YzypNr34itUfSvYJcv+eYdTnTvOZ2vD3cA4=
github.com/go-git/go-git/v5 v5.11.0/go.mod h1:6GFcX2P3NM7FPBfpePbpLd21XxsgdAt+lKqXmCUiUCY=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
@ -118,6 +124,7 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/unascribed/FlexVer/go/flexver v1.0.0 h1:eaAAWwaT8TiGK75wfEgQRPRVJc1ZIiLTLGUKXxpcs0c=
github.com/unascribed/FlexVer/go/flexver v1.0.0/go.mod h1:OkWZGfmV3DV2ADlgoS7W1+dD1OOci4mEracZCi3ulBk=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
@ -132,12 +139,16 @@ golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc=
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/crypto v0.16.0 h1:mMMrFzRSCF0GvB7Ne27XVtVAaXLrPmgPC7/v0tkwHaY=
golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
golang.org/x/crypto v0.19.0 h1:ENy+Az/9Y1vSrlrvBSyna3PITt4tiZLf7sgCjZBX7Wo=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0=
golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.15.0 h1:SernR4v+D55NyBH2QiEQrlBAnj1ECL6AGrA5+dPaMY8=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
@ -152,6 +163,8 @@ golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c=
golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U=
golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -176,6 +189,8 @@ golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc=
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.17.0 h1:25cE3gD+tdBA7lp7QfhuV+rJiE9YXTcS3VG1SqssI/Y=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
@ -186,6 +201,8 @@ golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek=
golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4=
golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0=
golang.org/x/term v0.17.0 h1:mkTF7LCd6WGJNL3K1Ad7kwxNfYAW6a8a8QqtMblp/4U=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@ -206,6 +223,10 @@ golang.org/x/tools v0.14.0 h1:jvNa2pY0M4r62jkRQ6RwEZZyPcymeL9XZMLBbV7U2nc=
golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg=
golang.org/x/tools v0.16.0 h1:GO788SKMRunPIBCXiQyo2AaexLstOrVhuAL5YwsckQM=
golang.org/x/tools v0.16.0/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0=
golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc=
golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps=
golang.org/x/tools v0.18.0 h1:k8NLag8AGHnn+PHbl7g43CtqZAwG60vZkLqgyZgIHgQ=
golang.org/x/tools v0.18.0/go.mod h1:GL7B4CwcLLeo59yx/9UWWuNOW1n3VZ4f5axWfML7Lcg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

View File

@ -7,10 +7,12 @@ package project
import (
"crypto/sha256"
"database/sql"
"errors"
"fmt"
"log"
"sort"
"strings"
"sync"
"time"
"github.com/unascribed/FlexVer/go/flexver"
@ -21,6 +23,7 @@ import (
)
type Project struct {
ID string
URL string
Name string
Forge string
@ -29,30 +32,43 @@ type Project struct {
}
type Release struct {
ID string
URL string
Tag string
Content string
Date time.Time
ID string
ProjectID string
URL string
Tag string
Content string
Date time.Time
}
// GetReleases returns a list of all releases for a project from the database
func GetReleases(dbConn *sql.DB, proj Project) (Project, error) {
ret, err := db.GetReleases(dbConn, proj.URL)
func GetReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
proj.ID = GenProjectID(proj.URL, proj.Name, proj.Forge)
ret, err := db.GetReleases(dbConn, proj.ID)
if err != nil {
return proj, err
}
if len(ret) == 0 {
return fetchReleases(dbConn, proj)
proj, err = fetchReleases(dbConn, mu, proj)
if err != nil {
return proj, err
}
err = upsertReleases(dbConn, mu, proj.ID, proj.Releases)
if err != nil {
return proj, err
}
return proj, nil
}
for _, row := range ret {
proj.Releases = append(proj.Releases, Release{
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
ID: row["id"],
ProjectID: proj.ID,
Tag: row["tag"],
Content: row["content"],
URL: row["release_url"],
Date: time.Time{},
})
}
proj.Releases = SortReleases(proj.Releases)
@ -60,7 +76,7 @@ func GetReleases(dbConn *sql.DB, proj Project) (Project, error) {
}
// fetchReleases fetches releases from a project's forge given its URI
func fetchReleases(dbConn *sql.DB, p Project) (Project, error) {
func fetchReleases(dbConn *sql.DB, mu *sync.Mutex, p Project) (Project, error) {
var err error
switch p.Forge {
case "github", "gitea", "forgejo":
@ -71,13 +87,13 @@ func fetchReleases(dbConn *sql.DB, p Project) (Project, error) {
}
for _, release := range rssReleases {
p.Releases = append(p.Releases, Release{
ID: genReleaseID(p.URL, release.URL, release.Tag),
ID: GenReleaseID(p.URL, release.URL, release.Tag),
Tag: release.Tag,
Content: release.Content,
URL: release.URL,
Date: release.Date,
})
err = upsert(dbConn, p.URL, p.Releases)
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -90,13 +106,13 @@ func fetchReleases(dbConn *sql.DB, p Project) (Project, error) {
}
for _, release := range gitReleases {
p.Releases = append(p.Releases, Release{
ID: genReleaseID(p.URL, release.URL, release.Tag),
ID: GenReleaseID(p.URL, release.URL, release.Tag),
Tag: release.Tag,
Content: release.Content,
URL: release.URL,
Date: release.Date,
})
err = upsert(dbConn, p.URL, p.Releases)
err = upsertReleases(dbConn, mu, p.ID, p.Releases)
if err != nil {
log.Printf("Error upserting release: %v", err)
return p, err
@ -114,12 +130,18 @@ func SortReleases(releases []Release) []Release {
return releases
}
// upsert updates or inserts a project release into the database
func upsert(dbConn *sql.DB, url string, releases []Release) error {
func SortProjects(projects []Project) []Project {
sort.Slice(projects, func(i, j int) bool {
return strings.ToLower(projects[i].Name) < strings.ToLower(projects[j].Name)
})
return projects
}
// upsertReleases updates or inserts a release in the database
func upsertReleases(dbConn *sql.DB, mu *sync.Mutex, projID string, releases []Release) error {
for _, release := range releases {
date := release.Date.Format("2006-01-02 15:04:05")
id := genReleaseID(url, release.URL, release.Tag)
err := db.UpsertRelease(dbConn, id, url, release.URL, release.Tag, release.Content, date)
err := db.UpsertRelease(dbConn, mu, release.ID, projID, release.URL, release.Tag, release.Content, date)
if err != nil {
log.Printf("Error upserting release: %v", err)
return err
@ -128,34 +150,47 @@ func upsert(dbConn *sql.DB, url string, releases []Release) error {
return nil
}
func genReleaseID(projectURL, releaseURL, tag string) string {
// GenReleaseID generates a likely-unique ID from its project's URL, its release's URL, and its tag
func GenReleaseID(projectURL, releaseURL, tag string) string {
idByte := sha256.Sum256([]byte(projectURL + releaseURL + tag))
return fmt.Sprintf("%x", idByte)
}
func Track(dbConn *sql.DB, manualRefresh *chan struct{}, name, url, forge, release string) {
err := db.UpsertProject(dbConn, url, name, forge, release)
// GenProjectID generates a likely-unique ID from a project's URI, name, and forge
func GenProjectID(url, name, forge string) string {
idByte := sha256.Sum256([]byte(url + name + forge))
return fmt.Sprintf("%x", idByte)
}
func Track(dbConn *sql.DB, mu *sync.Mutex, manualRefresh *chan struct{}, name, url, forge, release string) {
id := GenProjectID(url, name, forge)
err := db.UpsertProject(dbConn, mu, id, url, name, forge, release)
if err != nil {
fmt.Println("Error upserting project:", err)
}
*manualRefresh <- struct{}{}
}
func Untrack(dbConn *sql.DB, manualRefresh *chan struct{}, url string) {
err := db.DeleteProject(dbConn, url)
func Untrack(dbConn *sql.DB, mu *sync.Mutex, id string) {
proj, err := db.GetProject(dbConn, id)
if err != nil {
fmt.Println("Error getting project:", err)
}
err = db.DeleteProject(dbConn, mu, proj["id"])
if err != nil {
fmt.Println("Error deleting project:", err)
}
*manualRefresh <- struct{}{}
err = git.RemoveRepo(url)
// TODO: before removing, check whether other tracked projects use the same
// repo
err = git.RemoveRepo(proj["url"])
if err != nil {
log.Println(err)
}
}
func RefreshLoop(dbConn *sql.DB, interval int, manualRefresh, req *chan struct{}, res *chan []Project) {
func RefreshLoop(dbConn *sql.DB, mu *sync.Mutex, interval int, manualRefresh, req *chan struct{}, res *chan []Project) {
ticker := time.NewTicker(time.Second * time.Duration(interval))
fetch := func() []Project {
@ -164,7 +199,7 @@ func RefreshLoop(dbConn *sql.DB, interval int, manualRefresh, req *chan struct{}
fmt.Println("Error getting projects:", err)
}
for i, p := range projectsList {
p, err := fetchReleases(dbConn, p)
p, err := fetchReleases(dbConn, mu, p)
if err != nil {
fmt.Println(err)
continue
@ -175,7 +210,7 @@ func RefreshLoop(dbConn *sql.DB, interval int, manualRefresh, req *chan struct{}
return strings.ToLower(projectsList[i].Name) < strings.ToLower(projectsList[j].Name)
})
for i := range projectsList {
err = upsert(dbConn, projectsList[i].URL, projectsList[i].Releases)
err = upsertReleases(dbConn, mu, projectsList[i].ID, projectsList[i].Releases)
if err != nil {
fmt.Println("Error upserting release:", err)
continue
@ -202,20 +237,33 @@ func RefreshLoop(dbConn *sql.DB, interval int, manualRefresh, req *chan struct{}
}
// GetProject returns a project from the database
func GetProject(dbConn *sql.DB, url string) (Project, error) {
projectDB, err := db.GetProject(dbConn, url)
if err != nil {
return Project{}, err
func GetProject(dbConn *sql.DB, proj Project) (Project, error) {
projectDB, err := db.GetProject(dbConn, proj.ID)
if err != nil && errors.Is(err, sql.ErrNoRows) {
return proj, nil
} else if err != nil {
return proj, err
}
p := Project{
URL: projectDB["url"],
Name: projectDB["name"],
Forge: projectDB["forge"],
ID: proj.ID,
URL: proj.URL,
Name: proj.Name,
Forge: proj.Forge,
Running: projectDB["version"],
}
return p, err
}
// GetProjectWithReleases returns a single project from the database along with its releases
func GetProjectWithReleases(dbConn *sql.DB, mu *sync.Mutex, proj Project) (Project, error) {
project, err := GetProject(dbConn, proj)
if err != nil {
return Project{}, err
}
return GetReleases(dbConn, mu, project)
}
// GetProjects returns a list of all projects from the database
func GetProjects(dbConn *sql.DB) ([]Project, error) {
projectsDB, err := db.GetProjects(dbConn)
@ -226,6 +274,7 @@ func GetProjects(dbConn *sql.DB) ([]Project, error) {
projects := make([]Project, len(projectsDB))
for i, p := range projectsDB {
projects[i] = Project{
ID: p["id"],
URL: p["url"],
Name: p["name"],
Forge: p["forge"],
@ -233,5 +282,24 @@ func GetProjects(dbConn *sql.DB) ([]Project, error) {
}
}
return projects, nil
return SortProjects(projects), nil
}
// GetProjectsWithReleases returns a list of all projects and all their releases
// from the database
func GetProjectsWithReleases(dbConn *sql.DB, mu *sync.Mutex) ([]Project, error) {
projects, err := GetProjects(dbConn)
if err != nil {
return nil, err
}
for i := range projects {
projects[i], err = GetReleases(dbConn, mu, projects[i])
if err != nil {
return nil, err
}
projects[i].Releases = SortReleases(projects[i].Releases)
}
return SortProjects(projects), nil
}

View File

@ -18,19 +18,51 @@
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body>
<h1>Willow &nbsp;&nbsp;&nbsp;<span><a href="/logout">Log out</a></span></h1>
<p><a href="/new">Track a new project</a></p>
<div class="projects">
<!-- Range through projects that aren't yet up-to-date -->
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<div class="project">
<h2><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h2>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
<p>Latest: <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a></p>
<p>
<details>
<summary>Expand release notes</summary>
<header class="wrapper">
<h1>Willow &nbsp;&nbsp;&nbsp;<span><a href="/logout">Log out</a></span></h1>
<p><a href="/new">Track a new project</a></p>
</header>
<div class="two_column">
<div class="projects">
<!-- Range through projects that aren't yet up-to-date -->
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<h2>Outdated projects</h2>
{{- break -}}
{{- end -}}
{{- end -}}
{{- range . -}}
{{- if ne .Running (index .Releases 0).Tag -}}
<div id="{{ .ID }}" class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
<p>Latest: <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a></p>
<p><a href="#{{ (index .Releases 0).ID }}">View release notes</a></p>
</div>
{{- end -}}
{{- end -}}
<!-- Range through projects that _are_ up-to-date -->
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<h2>Up-to-date projects</h2>
{{- break -}}
{{- end -}}
{{- end -}}
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<div class="project card">
<h3><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span class="delete"><a href="/new?action=delete&id={{ .ID }}">Delete?</a></span></h3>
<p>You've selected <a href="#{{ (index .Releases 0).ID }}">{{ .Running }}</a>. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
</div>
{{- end -}}
{{- end -}}
</div>
<div class="release_notes">
<h2>Release notes</h2>
{{- range . -}}
<div id="{{ (index .Releases 0).ID }}" class="release_note card">
<h3>{{ .Name }}: release notes for <a href="{{ (index .Releases 0).URL }}">{{ (index .Releases 0).Tag }}</a> <span class="close"><a href="#">&#x2716;</a></span></h3>
{{- if eq .Forge "github" "gitea" "forgejo" -}}
{{- (index .Releases 0).Content -}}
{{- else -}}
@ -38,20 +70,10 @@
{{- (index .Releases 0).Content -}}
</pre>
{{- end -}}
</details>
</p>
<p><a class="return_to_project" href="#{{ .ID }}">Back to project</a></p>
</div>
{{- end -}}
</div>
</div>
{{- end -}}
{{- end -}}
<!-- Range through projects that _are_ up-to-date -->
{{- range . -}}
{{- if eq .Running (index .Releases 0).Tag -}}
<div class="project">
<h2><a href="{{ .URL }}">{{ .Name }}</a>&nbsp;&nbsp;&nbsp;<span style="font-size: 12px;"><a href="/new?action=delete&url={{ .URL }}">Delete?</a></span></h2>
<p>You've selected {{ .Running }}. <a href="/new?action=update&url={{ .URL }}&forge={{ .Forge }}&name={{ .Name }}">Modify?</a></p>
</div>
{{- end -}}
{{- end -}}
</body>
</html>

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body>
<body class="wrapper">
<h1>Willow</h1>
<form method="post">
<div class="input">

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body>
<body class="wrapper">
<h1>Willow</h1>
<form method="post">
<div class="input">

View File

@ -17,7 +17,7 @@
<link rel="preload" href="/static/styles.css" as="style" />
<link rel="stylesheet" href="/static/styles.css" />
</head>
<body>
<body class="wrapper">
<h1>Willow</h1>
<form method="post">
<div class="input">
@ -31,9 +31,9 @@
<label for="{{ .Tag }}"><a href="{{ .URL }}">{{ .Tag }}</a></label><br>
{{- else -}}
{{- if eq $forge "sourcehut" -}}
<label for="{{ .Tag }}"><a href="{{ $url }}/refs/{{ .Tag }}">{{ .Tag }}</label><br>
<label for="{{ .Tag }}"><a href="{{ $url }}/refs/{{ .Tag }}">{{ .Tag }}</a></label><br>
{{- else if eq $forge "gitlab" -}}
<label for="{{ .Tag }}"><a href="{{ $url }}/-releases/{{ .Tag }}">{{ .Tag }}</label><br>
<label for="{{ .Tag }}"><a href="{{ $url }}/-releases/{{ .Tag }}">{{ .Tag }}</a></label><br>
{{- else -}}
<label for="{{ .Tag }}">{{ .Tag }}</label><br>
{{- end -}}
@ -43,6 +43,7 @@
<input type="hidden" name="url" value="{{ .URL }}">
<input type="hidden" name="name" value="{{ .Name }}">
<input type="hidden" name="forge" value="{{ .Forge }}">
<input type="hidden" name="id" value="{{ .ID }}">
<input class="button" type="submit" formaction="/new" value="Track future releases">
</form>
<!-- Append these if they ever start limiting RSS entries: `(eq $forge "gitea") (eq $forge "forgejo")` -->

View File

@ -37,12 +37,11 @@
}
html {
max-width: 500px;
margin: auto auto;
color: #2f2f2f;
background: white;
font-family: 'Atkinson Hyperlegible', sans-serif;
padding-bottom: 20px;
scroll-behavior: smooth;
}
a {
@ -53,8 +52,28 @@ a:visited {
color: #0640e0;
}
.project {
max-width: 500px;
.two_column {
display: flex;
gap: 30px;
flex-direction: row;
margin: auto auto;
max-width: 1000px;
height: 92vh;
}
.projects, .release_notes {
overflow: scroll;
flex: 0 0 500px;
}
.release_note.card:not(:target) { display: none; }
.release_note.card:target { display: block; }
.return_to_project {
display: none;
}
.card {
border: 2px solid #2f2f2f;
background: #f8f8f8;
border-radius: 5px;
@ -63,36 +82,37 @@ a:visited {
box-shadow: 0 4px 8px 0 rgba(0,0,0,0.2),0 6px 20px 0 rgba(0,0,0,0.19);
}
.project > h2 {
.card > h3 {
margin-top: 0;
}
.project > p:first-of-type {
.card > p:first-of-type {
margin-bottom: 16px;
}
.project > h2 > span {
.card > p:last-of-type {
margin-bottom: 16px;
}
.close, .delete { float: right; }
.delete { font-size: 12px; }
.close > a {
text-decoration: none;
color: #2f2f2f;
}
.card > pre, .card > div > pre { overflow: scroll; }
.wrapper {
max-width: 500px;
margin: auto auto;
}
header > h1 > span {
font-size: 12px;
float: right;
}
body > h1 > span {
font-size: 12px;
float: right;
}
.project > details > pre {
overflow: scroll;
}
summary {
cursor: pointer;
}
details summary > * {
display: inline;
}
@media (prefers-color-scheme: dark) {
html {
background: #171717;
@ -107,8 +127,36 @@ details summary > * {
color: #5582ff;
}
.project {
border: 2px solid #ccc;
.card {
border: 2px solid #424242;
background: #1c1c1c;
}
.close > a {
color: #ccc;
}
}
@media only screen and (max-width: 1000px) {
div[id] {
display: block;
}
.two_column {
flex-direction: column;
}
.projects, .release_notes {
overflow: visible;
flex: 0 0 100%;
}
.return_to_project {
display: block;
}
.close {
display: none;
}
}

104
ws/ws.go
View File

@ -16,18 +16,17 @@ import (
"text/template"
"time"
"git.sr.ht/~amolith/willow/users"
"git.sr.ht/~amolith/willow/project"
"git.sr.ht/~amolith/willow/users"
"github.com/microcosm-cc/bluemonday"
)
type Handler struct {
DbConn *sql.DB
Mutex *sync.Mutex
Req *chan struct{}
ManualRefresh *chan struct{}
Res *chan []project.Project
Mu *sync.Mutex
}
//go:embed static
@ -41,8 +40,16 @@ func (h Handler) RootHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/login", http.StatusSeeOther)
return
}
*h.Req <- struct{}{}
data := <-*h.Res
data, err := project.GetProjectsWithReleases(h.DbConn, h.Mu)
if err != nil {
fmt.Println(err)
w.WriteHeader(http.StatusInternalServerError)
_, err := w.Write([]byte("Internal Server Error"))
if err != nil {
fmt.Println(err)
}
return
}
tmpl := template.Must(template.ParseFS(fs, "static/home.html"))
if err := tmpl.Execute(w, data); err != nil {
fmt.Println(err)
@ -73,7 +80,34 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
proj, err := project.GetProject(h.DbConn, submittedURL)
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj := project.Project{
ID: project.GenProjectID(submittedURL, name, forge),
URL: submittedURL,
Name: name,
Forge: forge,
}
proj, err := project.GetProject(h.DbConn, proj)
if err != nil && err != sql.ErrNoRows {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte(fmt.Sprintf("Error getting project: %s", err)))
@ -83,38 +117,7 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
if proj.Running == "" {
forge := bmStrict.Sanitize(params.Get("forge"))
if forge == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No forge provided"))
if err != nil {
fmt.Println(err)
}
return
}
name := bmStrict.Sanitize(params.Get("name"))
if name == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No name provided"))
if err != nil {
fmt.Println(err)
}
return
}
proj = project.Project{
URL: submittedURL,
Name: name,
Forge: forge,
}
proj.URL = strings.TrimSuffix(proj.URL, ".git")
}
proj, err = project.GetReleases(h.DbConn, proj)
proj, err = project.GetReleases(h.DbConn, h.Mu, proj)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte(fmt.Sprintf("Error getting releases: %s", err)))
@ -129,8 +132,8 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println(err)
}
} else if action == "delete" {
submittedURL := params.Get("url")
if submittedURL == "" {
submittedID := params.Get("id")
if submittedID == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No URL provided"))
if err != nil {
@ -139,7 +142,7 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
return
}
project.Untrack(h.DbConn, h.ManualRefresh, submittedURL)
project.Untrack(h.DbConn, h.Mu, submittedID)
http.Redirect(w, r, "/", http.StatusSeeOther)
}
}
@ -149,28 +152,29 @@ func (h Handler) NewHandler(w http.ResponseWriter, r *http.Request) {
if err != nil {
fmt.Println(err)
}
idValue := bmStrict.Sanitize(r.FormValue("id"))
nameValue := bmStrict.Sanitize(r.FormValue("name"))
urlValue := bmStrict.Sanitize(r.FormValue("url"))
forgeValue := bmStrict.Sanitize(r.FormValue("forge"))
releaseValue := bmStrict.Sanitize(r.FormValue("release"))
if nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue != "" {
project.Track(h.DbConn, h.ManualRefresh, nameValue, urlValue, forgeValue, releaseValue)
// If releaseValue is not empty, we're updating an existing project
if idValue != "" && nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue != "" {
project.Track(h.DbConn, h.Mu, h.ManualRefresh, nameValue, urlValue, forgeValue, releaseValue)
http.Redirect(w, r, "/", http.StatusSeeOther)
return
}
if nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue == "" {
// If releaseValue is empty, we're creating a new project
if idValue == "" && nameValue != "" && urlValue != "" && forgeValue != "" && releaseValue == "" {
http.Redirect(w, r, "/new?action=yoink&name="+url.QueryEscape(nameValue)+"&url="+url.QueryEscape(urlValue)+"&forge="+url.QueryEscape(forgeValue), http.StatusSeeOther)
return
}
if nameValue == "" && urlValue == "" && forgeValue == "" && releaseValue == "" {
w.WriteHeader(http.StatusBadRequest)
_, err := w.Write([]byte("No data provided"))
if err != nil {
fmt.Println(err)
}
w.WriteHeader(http.StatusBadRequest)
_, err = w.Write([]byte("No data provided"))
if err != nil {
fmt.Println(err)
}
}
}