Go Web Crawler

A simple web crawler written in Go that fetches the HTML content of a given URL.

Usage

To run the crawler, provide a starting URL as a command-line argument.

go run main.go <your-starting-url>

Example:

go run main.go https://example.com

You can build the project using the provided Makefile:

make build

This will create an executable named crawler and run it against a default URL.

To run the test suite:

make test

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.editorconfig		.editorconfig
.gitignore		.gitignore
.rgignore		.rgignore
Makefile		Makefile
README.md		README.md
crawl_page.go		crawl_page.go
extract_page_data.go		extract_page_data.go
extract_page_data_test.go		extract_page_data_test.go
get_content.go		get_content.go
get_content_test.go		get_content_test.go
get_html.go		get_html.go
get_images_from_html.go		get_images_from_html.go
get_images_from_html_test.go		get_images_from_html_test.go
get_urls_from_html.go		get_urls_from_html.go
get_urls_from_html_test.go		get_urls_from_html_test.go
go.mod		go.mod
go.sum		go.sum
main.go		main.go
normalize_url.go		normalize_url.go
normalize_url_test.go		normalize_url_test.go