Rework of DataSource and TileReader #11
Conversation
… test data, I will create a test later.
|
I think you have a namespace issue with your last commit as it is failing to build on the CI at the moment. |
|
Thank you, you are right. I renamed some projects and didn't test the benchmarks. I changed this. |
|
I had a good look and it all makes sense. My only concern would be that we need to align the interfaces that the AliFlux renderer uses. There is some cross over for sure and it would be nice if they could all use the same interface. This will then cause a few challenges - the AliFlux code defines its own VectorTile, and as discussed else where, the NTS VectorTile is different (or at least AliFlux at some point fixes up the coordinates.) What I'd like to do is get my current changes in, as they simplify the AliFlux renderer, and then we can rebase your changes and rework them to make as much of the code as common as we can. One comment more - we need to probably make sure the projects are configured. At the moment I ams supporting dotnetstandard2.0, net8.0 and net9.0 because I am supporting some codebases that still require those versions. If we can at all make 8.0 a target too, and attempt to make standard2.0 if possible, that would be good. I hope to drop standard support this year, but now quite there yet. |
|
Do you look into the sample app of my VectorTileLayer? It should run out of the box. There is, what you want to have. But it is a little complicated. And it isn't ready. The symbols, labels and street names missing. |
|
Up to now, I made no changes to your code. I only added more projects and some interfaces to your part. At the end, the old code could be deleted. Regarding the NTS coordinate problem, I made this PR. There we have to wait until it is accepted and moved on to NuGet. But then I would use the NTS implementation, because it is well tested and used by others and Mapsui use NTS already. Further it seems no problem to convert all projects to dotnetstandard2.0 and .net8.0. I did this already. I will not reuse any code from the AliFlux renderer. So I'm not interested in any part or what it uses. At the end, the renderer get a NTS VectorTile and a VectorStyle and produces from this a tile. |
|
I'm still not sure, if the name VectorTileReader is good. Is it clear to you, what it does? Or is the name VectorTileConverter better? It converts a byte[] to a VectorTile. |
Okay - that is perfect. Thank you. For my part, I like the ideas you have and I would like to make the AliFlux code comply to your interfaces where possible - or at least follow a common API style. So my plan is to make that happen as part of the transition to your code. I will resolve the conflicts you currently have by refactoring master to use as much of your rework as I can for the AliFlux side, but I will leave your main code in this PR so that it can later be merged. I don't think we will delete AliFlux any time in the near future, but I would like to have both "engines" available till we have stability.
Yes, it would be great if they could generate the tiles with either absolute or relative coordinates. We can use both, but not having to transform everything would make like a lot simpler.
Yes, it is a short term thing. I use this code in projects that support mainly 8.0 and a few with legacy old frameworks (a few 4.8 and one Xamarin Forms that will never be converted because it is for testing internally) and so at the moment I need all 3 targets. For UI apps, I'm trying to use Avalonia as far as possible because it is reasonably cross platform and I am doing 90% of the development on an Apple Silicon MacBook Pro, so the best I can get Windows 11 is in a VM that is running the ARM version. I will set up an Intel machine at some point, but I haven't had the time recently.
Yes, that is fair. I will be concentrating on the AliFlux in the short term as it is being used in production and it has those two big rendering flaws I would like to fix. That way we will not be stepping on each other's toes while you are getting the NTS part to sa stable point. I also think we can look at getting NTS to render pure vectors to the map surface in the future. It for sure works, but it is not viable without better accelerated rendering. |
|
Ok, I worked the last two days on a branch, that relays on this one. I added the possibility to read a Mapbox style file. I had to change the names of some parts (not one of your parts). I added a first try for a renderer. There is some more work to convert the style layers of a Mapbox style file to SKPaints that could be used to draw. But I did most of this work before in my vector tile layers. So it shouldn't be very hard. Most problems I have always with finding good names ;-) You could find the branch here. |
Greatt! One of the things on the list of improvements overall if to get a better style reader. Do you think this is reusable or is it specific to your implementation? |
|
Okay - I looked at the Style stuff - yes I think it can be used. With regards to the raster tiles - I wonder if there is a way to combine these with in the vector tile package. It looks like the style defines a source, but in a pure offline scenario, I'm not sure that really scales well. It would be interesting to provide a hybrid approach, where everything is offline. Maybe if the vector background is rendered transparent in a hybrid scenario we can have a secondary raster mbtiles file to provide a background to composit to the tile? This could even be an independent layer entirely under the vector layer? It needs more thought though. |
|
No problem. If you draw the tile, you draw first the background and then the sources they are added in the order they appear in the style file. Normally the raster tile is drawn below the vector tile and only for lower zooms (world map). If the source doesn't return any data, it isn't drawn. |
|
If you look into the normal Mapbox style file, you see that the maxzoom for the raster layer stops at 6 and between 0 and 6 it vanishes. It is only used for this zoom levels, because there aren't to much data in the vector tiles. Or it is used for shading. Then it is above the vector map with opacity. Regarding the style file: is isn't a full implementation. What is still missing are the Expressions. I started with them a time ago, but didn't made it complete ready. The Filter part should be incorporated. But beside this, it should work. Regarding the renderer I'm not sure, if I could make it totally independent of the style file type. But I try. The idea is to have different renderers for SKBitmaps or SKPictures independent from the style file type and the sources. I think the foundation for this is good this time. What I don't have is something to load things from a web address or embedded things. This is for loading style files, sprites and fonts. And also caching. When you read the code, is it for you understandable and logical? Do you need more comments? Are the names well selected? |
Okay - I think loading that data is quite trivial. As you said, BruTile can do this already. I think allowing an offline raster under layer would also be cool. That way you might have satellite imagery with a sunset of the vector data overlaid. That is for sure something we could support.
Sure. I will look at using it when you are happy. The AliFlux version is very messy at the moment and I was going to refactor it. But if we have a common implementation, there is no need to duplicate this work.
I think having AliFlux emit SKBitmap or SKPicture and then have an extension to make the PNG is probably giving library authors more options. For AliFlux, I think (as @pauldendulk suggested) abstracting the Skia code form the renderer would be helpful. Because we can then replace the ICanvas implementation more easily, and we can also experiment with other rendering libraries. ImageSharp comes to mind, as does VectSharp. But I have nothing against Skia so I'm not in a hurry to replace it.
AliFlux solves the fonts by having them as embedded resources. If you need similar fonts, maybe those can be moved to a common location? You might then only need to load a subset of the fonts. I think fonts in general are hard, and we would need a way to fallback sensibly to an available font on the local device/OS.
I'm all in favour of comments. I have no problem with your code. I do have a style I want to enforce, but I haven't committed the editor config yet. I do tend to gravitate towards C#11/12 style. Any suggestions I might make would be to follow those coding conventions. The main issue I had had is the relative paths in some of the test and benchmarks - under MacOS those paths seem to not work. Might be better to do what I was doing myself, and make the files linked to the project and push them to the output directory. It means they are local to the output soe it makes like a lot simpler. |
|
Hi, have you made any further progress? I'm planning one more fix to the AliFlux rendering this week and then I can start to look at how we can merge your code in. |
|
Yes, I made some progress. I'm ready with the style file loader, the sprite loader, the Mapbox SKPaint providers and the renderer for lines and polygons. Up to now I used NewtonSoft, NTS and SkiaSharp in some of the projects. I made some tests and benchmarks. I'm still missing the part, where Mapbox sources are converted to tile data sources. Perhaps I have time tomorrow. But I saw, that Paul is also working at this project. So it would be the best for your project when I leave. Better going now than thrown out in the future. I wish you and your project the best. |
Okay that is great news.
Paul is working on the AliFlux side of things at the moment. I think there is room for everyone. It's my project and I want to make the best vector renderer we can make. I appreciate your work and I want us to integrate it in to the repo when we can make something that is viable, and to ultimately replace AliFlux as the focus in the future. Paul is here because he and I have a common goal to create a vector renderer to be used by Mapsui and other C# mapping libraries in the future. I think you want this too. There is room for everyone and I appreciate your work. I would never want to push you out and I would like you to continue on with your branch because it will ultimately become a better option when it matures.
That is sad. I would rather have you here, but you need to make your own decision. |
|
@charlenni I forked your code on my second account as that will mean I can look at it properly and not have to deal with the merge conflicts for now. I will have a go at fixing the merge conflicts and do a PR in to your code. That is probably cleaner. If you do want to abandon this code, I will probably clean it up still and put in in to the main repo. Either way, you are a welcome contributor and always happy to have you as part of the project. |
|
I have started to look at this. I have a branch based on your Styler branch on my @emsonm second account (this is because I didn't want to deal with having a fork of the code and the original repo - this is cleaner for me.) I made a few changes - I introduces a I'm not done and I still need to merge master into it, but I think if I can get to the point where I am happy, we can work out which repo gets merged in to which repo and then back in to master. |
|
Where are the divergences between main and this branch? I thought, I didn't touched any of your parts. |
Main has changed and it would be good to ensure the code that can be shared (mostly data access) is shared. There are a few merge conflicts. It probably is trivial to fix. I will try to have a look tomorrow but I didn't get a chance to follow up today. |
|
Work still goes on. New branch is "symbols". I started with handling the symbols. Symbols are now separated from fills and lines. So they could be drawn independently of the underlaying map. For this, the symbol could check with a tree, if there is still space on the map, and, if yes, draw parts of the symbols and add itself to the tree for the next symbol. Have to see, if this will work. |
2 participants
Hello,
us discussed before, I implemented a DataSource for a MBTiles SQLite file. With this, it shouldn't be a problem to implement a file structure source or a HTTP source. Most of the code is grabbed from BruTile and adjusted a little.
Second I implemented a VectorTileReader. It is only a frame around the NetTopologySuite reader. With this, it shouldn't be very difficult to work with data in other vector tile formats. It uses a DataSource (which is for retriving data from a source) and converts this data (which is in GZipped PBF format) from Mapbox format to a NetTopologySuite VectorTile. This now could be used to access the layers and features of the vector data.
Added some basic test for this two things.
I hope this helps.