Complete Merge

.gitignore

@@ -1,2 +1,18 @@
 # Ignore all temporary directories created by Bazel.
-bazel-*
+bazel-*
+# Ignore local configs
+.idea
+.DS_Store
+.clwb
+*.json
+*.csv
+instacart-market-basket-analysis/
+__pycache__
+math_opt
+logs/
+init.sh
+*pb.*
+*pb2.*
+gurobi.log
+tools/retail
+math_opt_benchmark/retail

README.md

@@ -6,3 +6,103 @@ Unlike the traditional MIPLIB benchmarks, these benchmarks focus on incremental
 solves, decompositions, and using callbacks.
 
 This is not an officially supported Google product.
+
+##
+Required changes to MathOpt: math_opt.h
+```c++
+  ModelUpdateProto ExportModelUpdate() const {
+    return *(update_tracker_->ExportModelUpdate());
+  }
+```
+For me, the file is located in
+```shell script
+/private/var/tmp/_bazel_jeremyweiss/4b8ef44886ebc6aef2f85c5cac367848/external/com_google_ortools/ortools/math_opt/cpp/math_opt.h
+```
+
+## Installation
+Bazel will download all the C++ dependencies at build-time (see [WORKSPACE](WORKSPACE) for the full list). Everything was tested using v3.2.0.
+
+Python3 is required to generate random graphs, but it is not needed to run any of the
+benchmarks. The only dependency is NumPy.
+
+Gurobi needs to be installed manually. The instructions [from their website](https://www.gurobi.com/documentation/9.1/quickstart_mac/software_installation_guid.html)
+should cover everything.
+
+## Creating Random Instances
+<strong> Skip this section if you plan to use the included protos. </strong> The retail data is generated
+from the Instacart dataset available [here](https://www.kaggle.com/c/instacart-market-basket-analysis)
+and requires a Kaggle account to download.
+
+1. Random graphs:
+    ```
+    cd ./tools/graphs/
+    python3 ./generate_graphs.py
+    cd ../..
+    ```
+2. Instacart orders: the Instacart dataset must be in `./tools/retails/` and have its default name.
+    ```shell script
+    cd ./tools/retail/
+    python3 ./extract_orders.py
+    python3 ./format_data.py
+    cd ../..
+    ```
+
+## Generating the Iterative Updates
+From the home directory, run `bazel build -c opt ...` to build all of the programs. 
+The commands needed to generate the model updates for iterative solves are as follows:
+1. Minimum Spanning Tree:
+    ```shell script
+    export GRAPHDIR="./tools/graphs/graph_protos/"
+    export MST_OUTDIR="./protos/mst/"
+   ./bazel-bin/math_opt_benchmark/mst/mst_main --graph_dir=$GRAPHDIR --out_dir=$MST_OUTDIR | egrep -v "Academic license"
+    ```
+2. Uncapacitated Facility Location: <br> ORLIB contains the solutions to each instance, so we check correctness by comparing
+our output to theirs.
+    ```shell script
+    export UFL_OUTDIR="./math_opt_benchmark/facility/protos/"
+    for file in `cat ./tools/facility/filepaths.txt`; do
+      ./bazel-bin/math_opt_benchmark/facility/ufl_main --filename=$file --out_dir=$UFL_OUTDIR | egrep -v "Academic license" | diff $file.opt -
+    done
+    ```
+3. Split Shipments with Substitutions (not updated)
+   ```shell script
+   export CARTDIR="./tools/retail/dataset/"
+   export SUB_OUTDIR="./protos/subs/"
+   ./bazel-bin/math_opt_benchmark/retail/split_and_sub/split_and_sub_main --data_dir=$CARTDIR --solver_type=gurobi --solve_directly  | egrep -v "Academic license"
+   ```
+4. Cutting Stock
+   ```shell script
+   export STOCKDIR="./cutting_stock_data/fiber/"
+   export STOCK_OUTDIR="./protos/cutting_stock/"
+   for file in `ls $STOCKDIR`; do
+     ./bazel-bin/math_opt_benchmark/cutting_stock/cutting_stock_main --data=$STOCKDIR/$file --out_dir=$STOCK_OUTDIR
+   done
+   ```
+
+### Running the Benchmarks
+The script `./tools/benchmark.sh` will compare solver performance on all of the problems above, assuming default locations.
+```shell script
+bash ./tools/benchmark.sh
+```
+Every file in the `--instance_dir` directory will be run in the benchmark, so the only files in the directory should
+be text protos of [BenchmarkInstance](math_opt_benchmark/proto/model.proto).
+
+To benchmark individual models, add the `--is_single_file` flag
+```shell script
+./bazel-bin/math_opt_benchmark/benchmarker/benchmark_main --instance_dir=./protos/mst/1.txt --is_mip --is_single_file
+```
+
+### Directories
+[math_opt_benchmark/benchmarker](math_opt_benchmark/benchmarker) contains the main program which reads the BenchmarkInstance
+protos and compares solver performances. The protos are generated from the programs in the other
+directories:
+1. [math_opt_benchmark/facility](math_opt_benchmark/facility) contains the files to solve UFL
+    * [knapsack.cc](math_opt_benchmark/facility/knapsack.cc) solves the worker dual
+2. [math_opt_benchmark/mst](math_opt_benchmark/mst) contains the files to solve MST
+    * [graph](math_opt_benchmark/mst/graph) contains the max-flow separation oracle and
+    various graph algorithms for verifying the correctness of the MST.
+3. [math_opt_benchmark/retail](math_opt_benchmark/retail) contains the shipment models
+    * Not updated yet: [split](math_opt_benchmark/retail/split) minimizes split shipments
+    * [split_and_sub](math_opt_benchmark/retail/split_and_sub) minimizes split shipments with personalized substitutions
+4. [math_opt_benchmark/proto](math_opt_benchmark/proto) contains the protos used to store 
+[graphs](math_opt_benchmark/proto/graph.proto), [Instacart orders](math_opt_benchmark/proto/dataset.proto), and the [BenchmarkInstance](math_opt_benchmark/proto/model.proto)

math_opt_benchmark/benchmarker/BUILD

@@ -0,0 +1,20 @@
+licenses(["notice"])
+
+package(
+    default_visibility = [
+        "//visibility:public",
+    ],
+)
+
+cc_binary(
+    name = "benchmark_main",
+    srcs = ["benchmark_main.cc"],
+    deps = [
+        "//math_opt_benchmark/proto:model_cc_proto",
+        "@com_google_absl//absl/strings",
+        "@com_google_ortools//ortools/math_opt/cpp:math_opt",
+        "@com_google_ortools//ortools/math_opt/solvers:gscip_solver",
+        "@com_google_ortools//ortools/math_opt/solvers:gurobi_solver",
+        "@com_google_ortools//ortools/math_opt/solvers:glop_solver",
+    ],
+)

math_opt_benchmark/benchmarker/benchmark_main.cc

@@ -0,0 +1,142 @@
+// Copyright 2020 The MathOpt Benchmark Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <fstream>
+
+#include "absl/flags/flag.h"
+#include "absl/time/time.h"
+#include "ortools/math_opt/cpp/math_opt.h"
+#include "math_opt_benchmark/proto/model.pb.h"
+#include "google/protobuf/text_format.h"
+
+#define OUT
+ABSL_FLAG(std::string, instance_dir, "", "Path to directory containing model protos");
+ABSL_FLAG(bool, is_mip, true, "Path to directory containing model protos");
+ABSL_FLAG(bool, is_single_file, false, "Whether instance_dir points to a directory or a single file");
+
+namespace math_opt = operations_research::math_opt;
+
+const std::vector<math_opt::SolverType> lp_solvers{
+  math_opt::SolverType::kGlop,
+  math_opt::SolverType::kGurobi,
+//  math_opt::SolverType::kGlpk,
+};
+const std::vector<math_opt::SolverType> mip_solvers{
+    math_opt::SolverType::kGscip,
+    math_opt::SolverType::kGurobi,
+//    math_opt::SolverType::kGlpk,
+};
+
+
+namespace math_opt_benchmark {
+
+std::unique_ptr<BenchmarkInstance> LoadInstance(const std::string& filename) {
+  std::ifstream file(filename);
+  std::stringstream buffer;
+  buffer << file.rdbuf();
+  std::string proto_str = buffer.str();
+  BenchmarkInstance new_instance;
+  CHECK(google::protobuf::TextFormat::ParseFromString(proto_str, &new_instance));
+  return std::make_unique<BenchmarkInstance>(new_instance);
+}
+
+absl::Duration SolveModel(std::unique_ptr<BenchmarkInstance>& instance, std::unique_ptr<math_opt::Model>& model, math_opt::SolverType solver_type) {
+  absl::Time start_t = absl::Now();
+  std::unique_ptr<math_opt::IncrementalSolver> solver = math_opt::IncrementalSolver::New(*model, solver_type).value();
+
+  absl::StatusOr<math_opt::SolveResult> result = solver->Solve();
+  CHECK_EQ(result.value().termination.reason, math_opt::TerminationReason::kOptimal) << result.value().termination.detail;
+
+  // TODO store solve_stats?
+
+  double obj = result.value().objective_value();
+  CHECK_NEAR(obj, instance->objectives(0), 0.005);
+
+  for (int i = 0; i < instance->model_updates_size(); i++) {
+    CHECK_OK(model->ApplyUpdateProto(instance->model_updates(i)));
+    absl::StatusOr<math_opt::SolveResult> solution = solver->Solve();
+    CHECK_EQ(solution.value().termination.reason, math_opt::TerminationReason::kOptimal) << solution.value().termination.detail;
+    absl::Duration t = solution.value().solve_stats.solve_time;
+    obj = solution.value().objective_value();
+    CHECK_NEAR(obj, instance->objectives(i+1), 0.005);
+  }
+
+  absl::Duration elapsed = absl::Now() - start_t;
+
+  return elapsed;
+
+}
+
+absl::Duration max_t(const std::vector<absl::Duration>& v) {
+  absl::Duration max = absl::ZeroDuration();
+  for (absl::Duration t : v) {
+    if (max < t) {
+      max = t;
+    }
+  }
+
+  return max;
+}
+
+absl::Duration average_t(const std::vector<absl::Duration>& v) {
+  int64_t sum = 0;
+  for (absl::Duration t : v) {
+    sum += t / absl::Nanoseconds(1);
+  }
+  sum /= v.size();
+  return absl::Nanoseconds(sum);
+}
+
+void BenchmarkMain(const std::vector<std::string>& proto_filenames, const std::vector<math_opt::SolverType>& solvers) {
+  std::vector<std::vector<absl::Duration>> solve_times(solvers.size(), std::vector<absl::Duration>(proto_filenames.size()));
+  for (const std::string& filename : proto_filenames) {
+    std::unique_ptr<BenchmarkInstance> instance = LoadInstance(filename);
+    const math_opt::ModelProto& initial_model = instance->initial_model();
+
+    for (int i = 0; i < solvers.size(); i++) {
+    math_opt::SolverType solver_type = solvers[i];
+    absl::StatusOr<std::unique_ptr<math_opt::Model>> new_model = math_opt::Model::FromModelProto(initial_model).value();
+    absl::Duration solve_time = SolveModel(instance, new_model.value(), solver_type);
+    solve_times[i].push_back(solve_time);
+    }
+  }
+
+  // TODO figure out what to print
+  for (int i = 0; i < solvers.size(); i++) {
+    std::cout << std::left << std::setw(20) << solvers[i] << absl::FormatDuration(average_t(solve_times[i]));
+    std::cout << std::endl;
+  }
+
+}
+
+} // namespace math_opt_benchmark
+
+int main(int argc, char *argv[]) {
+  google::InitGoogleLogging(argv[0]);
+  absl::ParseCommandLine(argc, argv);
+  std::string dir = absl::GetFlag(FLAGS_instance_dir);
+  std::vector<std::string> proto_filenames;
+  if (absl::GetFlag(FLAGS_is_single_file)) {
+    proto_filenames.push_back(dir);
+  } else {
+    for (const auto &entry : std::filesystem::directory_iterator(dir)) {
+      proto_filenames.push_back(entry.path());
+    }
+  }
+
+  bool is_mip = absl::GetFlag(FLAGS_is_mip);
+  const std::vector<math_opt::SolverType> solvers = is_mip ? mip_solvers : lp_solvers;
+
+  math_opt_benchmark::BenchmarkMain(proto_filenames, solvers);
+}

math_opt_benchmark/facility/ufl_main.cc

@@ -21,8 +21,8 @@
 
 #include <fstream>
 
-ABSL_FLAG(std::string, filename, "", "Path to ORLIB problem specification.");
-ABSL_FLAG(std::string, out_dir, "./", "Directory to save protos.");
+ABSL_FLAG(std::string, filename, "", "Path to ORLIB problem specification");
+ABSL_FLAG(std::string, out_dir, "./", "Directory to save protos");
 ABSL_FLAG(bool, iterative, true, "Solve iteratively");
 
 namespace math_opt_benchmark {

math_opt_benchmark/mst/BUILD

@@ -0,0 +1,45 @@
+licenses(["notice"])
+
+package(
+    default_visibility = [
+        "//visibility:public",
+    ],
+)
+
+cc_library(
+    name = "mst",
+    srcs = ["mst.cc"],
+    hdrs = ["mst.h"],
+    deps = [
+        "//math_opt_benchmark/mst/graph",
+        "//math_opt_benchmark/mst/matrix",
+        "//math_opt_benchmark/proto:model_cc_proto",
+        "//math_opt_benchmark/proto:graph_cc_proto",
+        "@com_google_absl//absl/strings",
+        "@com_google_ortools//ortools/math_opt/cpp:math_opt",
+        "@com_google_ortools//ortools/math_opt/solvers:gscip_solver",
+        "@com_google_ortools//ortools/math_opt/solvers:gurobi_solver",
+        "@com_google_ortools//ortools/math_opt/solvers:glop_solver",
+    ],
+)
+
+cc_test(
+    name = "mst_test",
+    srcs = [],
+    deps = [
+        ":mst",
+        "@com_google_googletest//:gtest",
+        "@com_google_googletest//:gtest_main",
+        "@com_google_ortools//ortools/linear_solver",
+    ],
+)
+
+cc_binary(
+    name = "mst_main",
+    srcs = ["mst_main.cc"],
+    deps = [
+        ":mst",
+        "@com_google_absl//absl/random",
+        "@com_google_ortools//ortools/linear_solver",
+    ],
+)

math_opt_benchmark/mst/graph/BUILD

@@ -0,0 +1,31 @@
+licenses(["notice"])
+
+package(
+    default_visibility = [
+        "//visibility:public",
+    ],
+)
+
+cc_library(
+    name = "graph",
+    srcs = ["graph.cc"],
+    hdrs = ["graph.h", "flow.h"],
+    deps = [
+        "//math_opt_benchmark/mst/matrix",
+        "@com_google_absl//absl/strings",
+        "@com_google_ortools//ortools/linear_solver",
+        "@com_google_ortools//ortools/graph:max_flow",
+        "@com_google_ortools//ortools/graph:minimum_spanning_tree",
+    ],
+)
+
+cc_test(
+    name = "graph_test",
+    srcs = [],
+    deps = [
+        ":graph",
+        "@com_google_googletest//:gtest",
+        "@com_google_googletest//:gtest_main",
+        "@com_google_ortools//ortools/linear_solver",
+    ],
+)