What does it mean to publish your scientific paper in 2020?

Content.

  • Functions of conferences.
  • Peer review quality.
  • Conference prefiltering.
  • Role of double-blind peer review.
  • Benefits to the community.
  • Benefits to authors.
  • Signals.

Part 1: Defining science and the function of conference publications.

Functions of conferences.

  1. Knowledge dissemination: “I want people to know about the new knowledge I discovered.” The conference promises a certain minimum level of attention one’s work gets. In other words, if you don’t publish at top conferences, nobody would read your work.
  2. Feedback: I would like people to check my results through the review process and discussion.
  3. Formal goodies: checking boxes, required to defend Ph.D., for tenure package, performance review, to put into grant report, etc.
  4. Certification. Publishing at CVPR is hard, therefore valuable.
  5. Reputation-building: Listing certain conferences on C.V. as a way of building one’s name as a scientist.
  6. Networking: meeting with peers, potential employers, etc.
  1. Prefiltering: time is limited, so we outsource the selection of what we are reading to the reviewers.
  2. Certification: time is limited, so we outsource the quality control and result check to the reviewers. We create a basic classifier: “If the paper is published by a top-conference, it is true.”
  3. Special case of certification: for people outside the field without the basic qualifications to select work that meets basic quality guarantees.
  4. Authors promise to answer our questions (symmetrical to “attention for the author from audience,” and audience gets the guarantee that questions about the work will be answered at the talk or poster session).

Peer review quality.

  • recent benchmark “Metric Learning Reality Check” by Kevin Musgrave et al. specifically lists tens of top conference-published papers, which contain unfair comparisons.
  • A Unifying Perspective on Neighbor Embeddings along the Attraction-Repulsion Spectrum” by Böhm et al. shows that widely cited and used method UMAP properties are an artifact of the implementation rather than what was published.
  • Study “On the Convergence of Adam and Beyond” by Reddi et al. shows that the convergence proof of the popular Adam optimizer is wrong. This doesn’t make Adam a bad optimizer (practice shows that it is good), but rather that such a flaw went unnoticed by reviewers.
  • Study “Cracking BING and Beyond” by Zhao et al. shows that the great results of the popular BING object proposal (CVPR2014 oral, >1000 citations in Google Scholar) are not because of the proposed objectness detector, but are results of a clever implementation-related hacking of the 0.5 IoU metric.
  • Moreover, keep in mind that, all of the above examples are about the best of the best: top conferences. The quality of the average paper and peer review from the low-tier is much, much worse.

Conferences are filters, lotteries, or some poor combination.

  • Reviewer thinks that the dataset paper is not for the conference, while an AC thinks it is: OpenReview.
  • Aaron Hertzmann on rebuttals, where instructions specify that, “the rebuttal is for addressing factual errors in the reviews and for answering specific questions posed by reviewers.” He writes,
  • There is a paper by Kenneth Ward Church, Emerging trends: Reviewing the reviewers (again) discussing the problems of conferences in a similar way as we have. Specifically, are reviews good enough at certification? Should we outsource all of the prefiltering functions to the reviewers? For both questions, answer he leans to is “rather not.”

Role of double-blind peer review.

Part 2: Anonymous preprints are a special case of delaying release until acceptance.

Benefits to the research community of non-anonymous preprints.

Benefits to the authors of non-anonymous preprints.

What is signaled by top conference acceptances?

  1. the ability to pay conference fees, either out-of-pocket or with grants, and/or in a country with a favorable exchange rate,
  2. the ability to write well, attentive advisor, or employment stability to wait a few review cycles,
  3. adequate computing or laboratory resources to perform the work,
  4. having the type of citizenship that would allow one to get a visa to the hosting country.

Conclusions.

--

--

--

Computer Vision researcher and consultant. Co-founder of Ukrainian Research group “Szkocka”.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Saturn Now Has 82 Known Moons — so Why Did We Only Get One?

Scientists Discover There Are no Limits to Growth and Immortality is Within Reach

I wanted to share an excerpt of my own manifestation.

How Einstein Proved that Atoms Existed. And How You will too. Part 1.2

Axial S-1 Club — AbCellera

The Celebration of Cyclone Dreaming (CCD) Part 9/15

Windwalkers Theo Jansen

How Do Peppered Moths Spend The Winter?

How Do Peppered Moths Spend The Winter?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dmytro Mishkin

Dmytro Mishkin

Computer Vision researcher and consultant. Co-founder of Ukrainian Research group “Szkocka”.

More from Medium

CV Series 2— Image Formation Part 1 (Theory)

MACLEAN WORKSHOP 2021 Segmentation of VHR EO images using Unsupervised Learning.

“Neural Structured Learning”

The DeepLab Family